Semantic Werks

Part of my research used the Object Constraint Language to provide a model query language. Knowing nothing about OCL a year ago, I am now getting to be more familiar with it. The OCL specification is pretty well written, although hardly suffused with useful examples. Consequently, I bought the OCL book by Jos Warmer and Anneke Klepper.

OCL is associated with the OMG model-driven architecture (MDA) initiative. Consequently, the book assumes that a world in which all software is created by manipulating diagrams (ok, not all models are diagrams, but most are) is a good and inevitable one.

First off, diagrams are not necessary in doing modeling, in fact may be counter-productive. The authors bring out the modeling maturity levels, where 1 is denigrated as ‘the lowest level of professional software development’, with specifications residing in the developer’s head, and level 5 being full use of models to develop systems.

There is no evidence that I am aware of that shows that software built in level 1 is more {maintainable, usable, functional, cheaper} than level 5. This hierarchy is entirely based on conjecture, and as such is a good example of over-reliance on begging the question (i.e., let’s assume that in the ideal world, software will be built with models, and then start looking for premises to support that conclusion).

It would be a good study to compare software built ad-hoc (level 1), versus software built using models. Certainly the former approach has by far the greater degree of industry acceptance, so on the basis of adoption, at least, the modeling approach has completely failed to make its case.

This is why I argue for moving our field from one of ‘building solutions better’ to ‘understanding what to do’. Software engineering started in 1968 at a NATO conference on how to handle the massive amounts of software code that was clearly going to be a big part of the future. It was here that the idea that software production should be more systematic and repeatable — more like building a bridge — was first put forth.

And that’s all to the good. However, what we do as academics is not building things. I mean, I’ve written code, but I’m sure it’s horrible and unreliable. But it’s not my central concern. My central concern is science, and as I understand it, that means exploring new ideas and testing those ideas in the form of theories. Most of what passes for software engineering research, however, is the building of new and unproven technologies or, worse, ideas for new technologies in the form of frameworks. It’s as if physics was only theoretical, and never experimental.

Tim Bray had a great post recently on the ‘software crisis’. That wasn’t his phrase but it is his subject.

I challenged his conclusions on the basis of poor information. I said:

@lemire too bad @timbray’s post is completely lacking in empirical data (consultants  don’t count).

To which Tim replied:

@alex77 Follow some of the links from my piece. Tons of data. Also, get any COO drunk & you’ll hear.

And I finished with:

@timbray anecdote is not the same as peer-reviewed data, sadly. Software field is light-years from even being able to have a “ClimateGate”

So what did I mean? And what did he mean?

Hacker science

There are a lot of Tims in the industry (and this is a good thing!). For example, his Wide Finder project is a classic example of ‘tinkering’ leading to innovation. I think the best way to characterize his work is engineering science. He starts with an itch, and tries to figure out the best way to scratch it. Along the way, we get new understanding of the problems (e.g., how can non-experts use Clojure effectively) and some data on what works and what doesn’t (in the ‘real world’). This is very similar to the development of the airfoil, and this stream of science is well described by Vincenti in “What Engineers Know and How They Know It”.

But this isn’t enough. Vincenti makes clear that further airfoil improvements come from a good understanding of the physics underlying lift and computer modeling, not iterative tinkering (although there are clear parallels). Similarly, while the Wide Finder project is a great way to start understanding the theories and constructs involved in concurrent programming in large-scale problems, I’m not convinced this is sufficient. What we would need is studies of many programmers working on many different problems. For example, how would the C solution work with non-expert C programmers? Do the concepts of Agents in Erlang make intuitive sense to most people?

Standard of evidence

These questions cannot be answered by individual tinkering. They require larger scale studies and more thorough investigation. But unfortunately, there is a trend in the software blogosphere to use one or two data points as solid evidence that “C sucks” or “Perl is unreadable”.

But why would we accept this standard of evidence for programming, but not for, say,  climate change? In the same way, can we really base conclusions about the state of IT on the musings of some influential bloggers, industry whitepapers, and self-serving consultant reports? Let’s imagine the same situation prevailed in climate science. Then we would have a few people commenting on the basis of anecdotal encounters (“boy, it sure is cold in Toronto this winter!”), Shell Oil writing reports on the oilsands (“only 100 hectares are polluted!”) and a climate mitigation consultant encouraging mitigation strategies. In none of these cases do you have enough information for rational action (not that this is always the goal!).

Maybe you would say these are apples and oranges, that climate science is a much bigger issue, etc. etc. I’m not sure. In both cases we are talking about billions of dollars of costs. The only real difference is that policy action will be taken by collective action (climate change) or corporate action (IT).

Improve the Data

So am I saying that Tim Bray is wrong, that corporate IT is fine, that the lessons of Twitter don’t apply? Not really. What I am saying is that you should base these sorts of conclusions on empirical data that is collected in the best traditions of open science. It would be data that is from a survey whose questions we can see. It would be data from peer-reviewed research journals.

Let’s stop making conclusions using data from Gartner, Standish, et al. These firms are in the business of selling advice, and they are not interested in objective truth. A lot of the reports are based on closed source surveys, talks with “Thought Leaders”, random observations, etc. If you are Standish, are you really interested in a report that says “Most IT projects successful”?

What is needed is a non-partisan study, much like the IPCC, that will examine the relevant scientific research on the issue. Before we can draw conclusions, especially black and white conclusions, we need to know what we don’t know (“unknown unknowns”!). This means raw data, and this means more openness on how well these IT projects are really doing. It would mean allowing researchers access to IT development teams, to perform proper case studies, to see, for example, why the FBI system failed. Too often software researchers are in the position of begging companies to release data. And even in industries where publicly accessible studies are mandated, we find many games being played to prevent negative outcomes being published.

So will there be improvements in empirical data on software? I have my doubts. But it’s necessary if we really want to know what software can do, and what software cannot do.