Semantic Werks

Tim Bray had a great post recently on the ‘software crisis’. That wasn’t his phrase but it is his subject.

I challenged his conclusions on the basis of poor information. I said:

@lemire too bad @timbray’s post is completely lacking in empirical data (consultants  don’t count).

To which Tim replied:

@alex77 Follow some of the links from my piece. Tons of data. Also, get any COO drunk & you’ll hear.

And I finished with:

@timbray anecdote is not the same as peer-reviewed data, sadly. Software field is light-years from even being able to have a “ClimateGate”

So what did I mean? And what did he mean?

Hacker science

There are a lot of Tims in the industry (and this is a good thing!). For example, his Wide Finder project is a classic example of ‘tinkering’ leading to innovation. I think the best way to characterize his work is engineering science. He starts with an itch, and tries to figure out the best way to scratch it. Along the way, we get new understanding of the problems (e.g., how can non-experts use Clojure effectively) and some data on what works and what doesn’t (in the ‘real world’). This is very similar to the development of the airfoil, and this stream of science is well described by Vincenti in “What Engineers Know and How They Know It”.

But this isn’t enough. Vincenti makes clear that further airfoil improvements come from a good understanding of the physics underlying lift and computer modeling, not iterative tinkering (although there are clear parallels). Similarly, while the Wide Finder project is a great way to start understanding the theories and constructs involved in concurrent programming in large-scale problems, I’m not convinced this is sufficient. What we would need is studies of many programmers working on many different problems. For example, how would the C solution work with non-expert C programmers? Do the concepts of Agents in Erlang make intuitive sense to most people?

Standard of evidence

These questions cannot be answered by individual tinkering. They require larger scale studies and more thorough investigation. But unfortunately, there is a trend in the software blogosphere to use one or two data points as solid evidence that “C sucks” or “Perl is unreadable”.

But why would we accept this standard of evidence for programming, but not for, say,  climate change? In the same way, can we really base conclusions about the state of IT on the musings of some influential bloggers, industry whitepapers, and self-serving consultant reports? Let’s imagine the same situation prevailed in climate science. Then we would have a few people commenting on the basis of anecdotal encounters (“boy, it sure is cold in Toronto this winter!”), Shell Oil writing reports on the oilsands (“only 100 hectares are polluted!”) and a climate mitigation consultant encouraging mitigation strategies. In none of these cases do you have enough information for rational action (not that this is always the goal!).

Maybe you would say these are apples and oranges, that climate science is a much bigger issue, etc. etc. I’m not sure. In both cases we are talking about billions of dollars of costs. The only real difference is that policy action will be taken by collective action (climate change) or corporate action (IT).

Improve the Data

So am I saying that Tim Bray is wrong, that corporate IT is fine, that the lessons of Twitter don’t apply? Not really. What I am saying is that you should base these sorts of conclusions on empirical data that is collected in the best traditions of open science. It would be data that is from a survey whose questions we can see. It would be data from peer-reviewed research journals.

Let’s stop making conclusions using data from Gartner, Standish, et al. These firms are in the business of selling advice, and they are not interested in objective truth. A lot of the reports are based on closed source surveys, talks with “Thought Leaders”, random observations, etc. If you are Standish, are you really interested in a report that says “Most IT projects successful”?

What is needed is a non-partisan study, much like the IPCC, that will examine the relevant scientific research on the issue. Before we can draw conclusions, especially black and white conclusions, we need to know what we don’t know (“unknown unknowns”!). This means raw data, and this means more openness on how well these IT projects are really doing. It would mean allowing researchers access to IT development teams, to perform proper case studies, to see, for example, why the FBI system failed. Too often software researchers are in the position of begging companies to release data. And even in industries where publicly accessible studies are mandated, we find many games being played to prevent negative outcomes being published.

So will there be improvements in empirical data on software? I have my doubts. But it’s necessary if we really want to know what software can do, and what software cannot do.

Vincenti, Walter, 1990. “What Engineers Know and How they Know It: analytical studies from aeronautical history”. Johns Hopkins University Press.

At RE08 in Barcelona, Michael Jackson mentioned this book as a must-read for people in requirements engineering, and respecting my elders, I dutifully picked it up.

The first contribution of this book (most important?) is to highlight that engineering (and I sort of unthinkingly lump software engineering in there, for the moment) is NOT just the application of scientific principles to real-world problems. They are ‘separate spheres of knowledge’.  Anyone who has tried to construct the most rudimentary technology — e.g., a tree-fort, change a tire, etc. can verify this. While on paper the solution is obvious, in harsh light of day, the nail doesn’t go in deep enough, the lugnuts are seized, and so on.

I found the book often went into excessive detail, of interest only to aeronautic buffs, but there were sections at the beginning and conclusion of each chapter that were highly interesting. Perhaps most relevant to my research, and to software engineering, was the chapter on aircraft qualities. Qualities include aspects like how ‘heavy’ the control stick is, how stable the plane flies, etc. Anyone who has driven several different types of car can relate to these properties, I would think: does the car shimmy at high speeds (like my first car, an 89 Toyota Tercel)? Do the brakes feel squishy? Does the door have a convincing thunk when closing, or a tinny clang?

For early aircraft design, these qualities were secondary to functional aspects: ‘With the essential proviso, of course, that an airplane can be reasonably and safely flown by the pilot, its utility depends crucially on these other characteristics [speed, range, ceiling, carrying capacity ...]‘. He goes on to add that only when these functional properties were largely satisfied (and here I assume he means that there were diminishing returns on further research into these functional characteristics) was flying quality considered. Eventually, specs for new aircraft came to include sections on flying qualities, encapsulating ‘the notion that desired subjective perceptions of pilots could be attained though objective specifications for designers’. Because these qualities were subjective, the specs tended to be design guides to assess tradeoffs rather than objective, measurable requirements (much like in NFRs and software design).

Vincenti argues that engineering knowledge grows via an evolutionary variation-selection process; as a result, ‘technology’s failures require examination as well as its successes’. The important lesson is that approaches that attempt to design something perfectly from the beginning seem doomed to failure. The best we can do is prepare to throw one version away, until we have a good body of evidence to support one particular solution. For example, NASA tries to re-use Mars expedition software and hardware as much as possible; e.g., the particular circuit board, the software language, and so on. The reason for this, as Jackson explained in his keynote, is to leverage the benefits from “Normal design” and specialization.The characteristics are known and well-tested; keeping these things constant allows for incremental specialization at the margins (e.g., improved science packages, new rocket engine, etc.)

My take-away is that the most important thing to understand in approaching a new problem is what pieces of that problem are normal and what pieces radical. Focusing on the radical is key, because we can assume this is where the failures and iteration will need to happen.