Semantic Werks

As part of my research, I try to keep up with the trends and tools in software development (it has nothing to do with procrastination, either).

I’m beginning to wonder if maybe I’m missing a large chunk of people in my reading. According to the blogs and sites I follow, agile is far superior to other methodologies, Ruby and Python are orders of magnitude more productive, REST architectures are preferred, and, importantly for me in particular, requirements are best understood as user stories, and not as lists of MUST and SHOULD items. Let’s call this the agile world.

But I feel like there’s this other world, the world I hear about when I read academic papers, or read IBM/Oracle/MSFT blogs, or hear about work at large companies. This is the world of J2EE, of COBOL, of UML models, of reams of paper documentation, of Rational Rose, of ESB, of SOA, of WS-*. This, for lack of a better term, is the waterfall world.

Is it just me, or is this waterfall world not getting the message that people like David Anderson and Martin Fowler are spreading? Is there a reason for this? Frequently argued is the point that these systems are orders of magnitude more complex, older, and larger, and therefore ‘agile’ methodologies just won’t work. And true, some of the agile proponents are tiresome ideologues who talk but don’t listen.

So while it is appealing to self-identify as the little boy pointing out the emperor’s lack of apparel, maybe I’m out of touch. Maybe there really is a need for complex modeling initiatives, for up-front requirements documentation. I’m not talking about safety-critical software either — that’s a whole different kettle of fish, and a straw-man for the purposes of this discussion. No, I’m specifically referring to large corporate IT departments, process control software, line automation, etc.

I guess I just don’t know how to experience the waterfall world without being involved in it first-hand. In the agile world, there is hype aplenty — hype sells books and seminars and certifications — but , at the risk of sounding repetitive, experience reports and anecdote just won’t cut it (for either side). It’s extraordinarily difficult in the software community to get at any objective source of information — except surveys, typically.

So who’s up for a PhD establishing the benefits of agile and its ecosystem over waterfall and its suite of tools? Can such a thing even be established?

Over on my research group’s wiki, I have created a quick list of possible ‘model problems’ for requirements engineering research. Contributions are welcome.

“Improving change tolerance through Capabilities-based design: an empirical analysis“, by Ramya Ravichandar, James D. Arthur, Shawn A. Bohner and David P. Tegarden. J. Softw. Maint. Evol.: Res. Pract. 2008; 20:135–170

The scope of this paper is building software systems that are change tolerant, where “the term ‘change tolerance’ connotes the ability of software to evolve within the bounds that it was designed”. It’s important to keep in mind the phrase “within the bounds” — this limits this technique to de novo design, rather than adaptive design (i.e., design that admits new components after the initial implementation). Any design that is designed to be change tolerant necessarily requires identification of potential changes up-front. And the authors acknowledge this with a small dig at agile techniques as being ‘unconventional’ (not anymore) and unproven (unlike the enormous body of evidence supporting waterfall processes .

It would seem the focus of the paper is on complex, emergent systems: “requirements and technology often evolve during the extended development periods of complex emergent systems, and thereby, inhibit a comprehensive up-front solution specification” (agreed); yet the empirical assessment is of a course evaluation system. They measure how many classes are impacted by each change to assess how CE reduces change impact.

There are several problems with their empirical assessment. First, they compare “RE-based design” with their technique. They obtain the RE-design by reverse engineering an existing system, but they have not validated it as a ‘good’ design (we need to compare apples and apples, right — both designs should have been created using the same amount of effort and skill). Second, the selection of change events is questionable; that the authors chose the change events implies bias in choosing those the CE-design is suited for. Finally, this system seems excessively trivial for a complex emergent system, as I mentioned before. Of course, it isn’t easy to evaluate truly complex systems.

Capability engineering is a combination of abstraction, reduced coupling, and high cohesion design (which really tells us nothing — who out there is designing systems that have high coupling and low cohesion? Possibly a few embedded systems, but not much else, I would wager). The key to the whole thing seems to be the functional decomposition, which is independent of system architecture/specification, and derived from user requirements. This commits the process fairly heavily to up-front requirements elicitiation, which is odd, since they claim to be concerned with complex, emergent systems, where (to my mind) requirements are never completely known from the beginning.

Complaints:

• no reference is made to feature modeling, which seems nearly identical;
• a bad case of EMS (Excessive Math Syndrome): too much detail about cohesion metrics, or algorithms for system design, when the real issue is coming up with the source numbers (anyone remember GIGO?);
• conclusion marred by the confounding factor that the RE-system might just be poorly designed;
• no reference to work on autonomic monitoring and correction (see Yiqiao‘s work).

Full marks:

• acknowledging the issue of change in software systems (although I’m biased)
• actually proposing a theory and testing it!
• discussing threats to validity

What I liked the most about this paper was its fairly scientific setup. I don’t agree that capability engineering is the panacea, but at least I can make concrete objections thanks to their well-structured paper.

On an unrelated Google expedition, I came across the ISBSSG, the International Software Standards Benchmarking Group. Their mission is to collect software and IT project data for comparison and metric purposes.

This seems like a great idea, and although I can’t find details, the fees for academic access are listed as ‘nominal’. Has anyone had experience with this group? I wonder what the data are like, in terms of representativeness, accuracy, usability.

Such a repository would provide some solution to the problem of data independence (e.g., not IBM-sponsored) and access.

On a somewhat related note: a recent study of cancer experiments showed that of the clinical trials registered with the FDA (which is mandatory), only 18% were reported in academic journals. Of the industry-sponsored trials, 75% were positive findings. While this seems bad, I bet the majority of unpublished studies were rejected by the peer-review process as not worth publishig. Still, the negative results ought to be published somewhere, even as a one-page report (via the excellent ‘Bad Science‘ blog).

Reading Robert Yin’s book on case studies has really opened my eyes. To date the most contentious aspect has been his distinction between statistical generalization and analytic generalization. The former is what is done in surveys and experiments: using a sample to make statements about the population. The latter is what he argues is done during a case study. A case study is not a sample, but rather part of an argument.

To properly frame the debate, I think it is important to start with the question of epistemology. That is, what is it we are seeking to do when we do research? In my opinion, it is to motivate the rational beliefs we can hold about a given observable phenomenon in the universe. For example, do requirements evolve? If they evolve, what is being done about it? Etc., etc.

Once we have a list of questions we are interested in — why is this apple hitting my head while the moon is always up in the sky? — we can begin to try and answer them. This is where the scientific method chosen is important. In phenomena whose causes can be controlled for, an experimental approach is useful. It allows us to generalize our results to a wider population. But a lot of phenomena in the world are not easily controllable. In this case, we need a technique that can allow us to still draw some conclusions about the phenomenon of interest. Otherwise, we would just throw up our hands in disgust. Is agile software development better than waterfall? Choosing a few good (representative) cases might well let us make some conclusions.

I think one of the important things to remember is that while case studies seem easy to do — just pick a subject and write up a history of the project — they are in reality much more complex to set up than their experimental cousins. This duality is what has led many to discount their scientific — epistemological — value. Most of what is called ‘case study’ in the literature is really more akin to ‘proof-of-concept’.