Semantic Werks

When we consider the nature of software-based systems over time, software evolution, there is always a lot of talk about evolving that system. Most famously, perhaps, a set of “Laws” were proposed regarding software evolution.* And to date, most work on software evolution tends to be very focused on technical issues:

• backward compatibility
• transformations
• versioning
• comparison

However, at heart this issue is not just one of technical challenges (low-hanging fruit, if you will). You are dealing with a change in domain fundamentals — e.g., a Car no longer has a combustion engine — and this forces a change in nearly all other layers of your technology. Whether you believe in Model-Driven Something in Upper Case or not, reality is that the domain model does drive your software-based systems: as soon as cars can now have battery engines, your existing software is no longer adequate.

Resolving the technical details, like updating your UML class models with new BatteryEngine boxes, is a simple enough idea, I think (obviously the devil is in the details). More of an issue is figuring out “what” the change means and “why” there was a change. This presents a challenge because we technology types would rather worry about the “how” since it is more amenable to immediate tinkering.

To answer the what-it-means and why-the-change questions (as well as impact analysis), I think there is only one answer: you must maintain the domain model in some explicit form for the duration of the software lifecycle. What form the model takes is context-dependent: I wouldn’t bother with a complex modeling solution if I was building a web site for a client, for example. It might be a DSL for a very specific set of business rules, or a massive set of finite state machines and a model checker. But you cannot answer questions about change without some explicit model artifact (unless you have some savant developer who maintains the domain model and implementation details in her/his brain over many years).

Ideally, this model will be able to answer the why and what questions for you automatically, or nearly automatically. It will explore scenarios, evaluate impacts, and allow you to choose the preferred evolution path (for some context-specific set of preferences). Even better, this model will evolve quickly, so that you can push changes to the software quickly and nearly transparently.

* The term “Laws” is more reflective of the modernist and logical positivist origins of software engineering, a paradigm that has long outlived its usefulness. The notion that nature can be controlled has been flung happily into the trash in most other disciplines.

This is part of the MSR Challenge series.

Having generated the lion’s share of the data I want to mine, I’ve turned my attention to getting interesting things out. Following a presentation to my research group, I got a few great tips on what they would like to see. One thing that came up was the notion of tying the data points (e.g., 2 mentions of ‘usability’ in 2004-March) to some external events. The idea is that there is a reason behind an increase in mentions/occurrences of the word/concept, beyond my (naïve) initial idea of “heightened interest” in a quality as the project goes on.

So the question becomes, how to extract external events that might be causing these spikes? We can readily imagine many such events: change in developer team, release cycle, source audits, and even more random things. Clearly, it would be impossible to determine all of these. But like a good historian, we must try to account for all the variability using some model of the system: like a biological organism, my contention is that the software reacts to the external events by modifying its phenotypic characteristics.

My first task is to determine what plausible external sources might be and account for these. After that, I am going to audit the frequency plots by looking at how much of the variability is accounted for by these ‘plausible’ sources. Anything remaining is an outlier that I will endeavour to explain (for a subset of the data).

The most obvious external forcing is release date. Particularly in Gnome, which has for a while [since when?] followed a coördinated release cycle, these fixed release dates are sure to prompt people to begin questioning software quality.

I tried to extract this information, at first, from subversion tags in the repository itself. Typically a release changeset gets tagged with the release id for future reference. However, the dataset is quite muddy (imports, inconsistencies, etc). I turned to the #gnome irc channel, and was pointed to the gnome-announce mailing list. There, each release is announced, even relatively minor ones. I should probably determine when this policy was not followed, but it seems reasonably safe to say that it is the rule, rather than the exception (and thus acceptable error). I wrote a little bash script to download (yay wget!) the mail archives (10 years worth), unzip them, and turn them into Apple mail folders. Now I can search by subject title for the project (‘evolution’) I want. I think I may also add these to my corpora (although they might work nicely as a control set).

“Improving change tolerance through Capabilities-based design: an empirical analysis“, by Ramya Ravichandar, James D. Arthur, Shawn A. Bohner and David P. Tegarden. J. Softw. Maint. Evol.: Res. Pract. 2008; 20:135–170

The scope of this paper is building software systems that are change tolerant, where “the term ‘change tolerance’ connotes the ability of software to evolve within the bounds that it was designed”. It’s important to keep in mind the phrase “within the bounds” — this limits this technique to de novo design, rather than adaptive design (i.e., design that admits new components after the initial implementation). Any design that is designed to be change tolerant necessarily requires identification of potential changes up-front. And the authors acknowledge this with a small dig at agile techniques as being ‘unconventional’ (not anymore) and unproven (unlike the enormous body of evidence supporting waterfall processes .

It would seem the focus of the paper is on complex, emergent systems: “requirements and technology often evolve during the extended development periods of complex emergent systems, and thereby, inhibit a comprehensive up-front solution specification” (agreed); yet the empirical assessment is of a course evaluation system. They measure how many classes are impacted by each change to assess how CE reduces change impact.

There are several problems with their empirical assessment. First, they compare “RE-based design” with their technique. They obtain the RE-design by reverse engineering an existing system, but they have not validated it as a ‘good’ design (we need to compare apples and apples, right — both designs should have been created using the same amount of effort and skill). Second, the selection of change events is questionable; that the authors chose the change events implies bias in choosing those the CE-design is suited for. Finally, this system seems excessively trivial for a complex emergent system, as I mentioned before. Of course, it isn’t easy to evaluate truly complex systems.

Capability engineering is a combination of abstraction, reduced coupling, and high cohesion design (which really tells us nothing — who out there is designing systems that have high coupling and low cohesion? Possibly a few embedded systems, but not much else, I would wager). The key to the whole thing seems to be the functional decomposition, which is independent of system architecture/specification, and derived from user requirements. This commits the process fairly heavily to up-front requirements elicitiation, which is odd, since they claim to be concerned with complex, emergent systems, where (to my mind) requirements are never completely known from the beginning.

Complaints:

• no reference is made to feature modeling, which seems nearly identical;
• a bad case of EMS (Excessive Math Syndrome): too much detail about cohesion metrics, or algorithms for system design, when the real issue is coming up with the source numbers (anyone remember GIGO?);
• conclusion marred by the confounding factor that the RE-system might just be poorly designed;
• no reference to work on autonomic monitoring and correction (see Yiqiao‘s work).

Full marks:

• acknowledging the issue of change in software systems (although I’m biased)
• actually proposing a theory and testing it!
• discussing threats to validity

What I liked the most about this paper was its fairly scientific setup. I don’t agree that capability engineering is the panacea, but at least I can make concrete objections thanks to their well-structured paper.