A common complaint in reviewing research papers is lack of real-world evaluation. But before we get to the thousands of requirements common in industry, we ought to also verify our approach with frequently used examples from the existing literature. I feel that ‘real-world’ is a substitute for voluminous; but often the problem is not (just) the sheer scale, but the interesting edge cases. For that job, I prefer we evaluate our work on smaller, easily understandable model problems.

The page is maintained at the software group’s web page, here.

My big take-away was his characterization of requirements as ‘perishable’. In his view, unmet requirements are like unsold inventory, essentially liabilities until they can be turned into working systems. He borrows from the just-in-time philosophy of Lean manufacturing to emphasize how unsatisfied requirements ought to be taken off the shelf and implemented as soon as possible.

In his view, future software/system architecture decisions will be much higher-level, business-value decisions. He mentioned software product lines as an example: the architect is someone who can separate product variability and encapsulate common behaviour, rather than over-specifying lower-level designs. There will be an increased need for ‘customer intimacy’ to generate unique systems that capture a core competency or competitive advantage.

There’s a lot more in this talk, including some interesting ideas on community evolution, so it’s well worth the time to watch.

The scope of this paper is building software systems that are change tolerant, where “the term ‘change tolerance’ connotes the ability of software to evolve within the bounds that it was designed”. It’s important to keep in mind the phrase “within the bounds” — this limits this technique to de novo design, rather than adaptive design (i.e., design that admits new components after the initial implementation). Any design that is designed to be change tolerant necessarily requires identification of potential changes up-front. And the authors acknowledge this with a small dig at agile techniques as being ‘unconventional’ (not anymore) and unproven (unlike the enormous body of evidence supporting waterfall processes .

It would seem the focus of the paper is on complex, emergent systems: “requirements and technology often evolve during the extended development periods of complex emergent systems, and thereby, inhibit a comprehensive up-front solution specification” (agreed); yet the empirical assessment is of a course evaluation system. They measure how many classes are impacted by each change to assess how CE reduces change impact.

There are several problems with their empirical assessment. First, they compare “RE-based design” with their technique. They obtain the RE-design by reverse engineering an existing system, but they have not validated it as a ‘good’ design (we need to compare apples and apples, right — both designs should have been created using the same amount of effort and skill). Second, the selection of change events is questionable; that the authors chose the change events implies bias in choosing those the CE-design is suited for. Finally, this system seems excessively trivial for a complex emergent system, as I mentioned before. Of course, it isn’t easy to evaluate truly complex systems.

Capability engineering is a combination of abstraction, reduced coupling, and high cohesion design (which really tells us nothing — who out there is designing systems that have high coupling and low cohesion? Possibly a few embedded systems, but not much else, I would wager). The key to the whole thing seems to be the functional decomposition, which is independent of system architecture/specification, and derived from user requirements. This commits the process fairly heavily to up-front requirements elicitiation, which is odd, since they claim to be concerned with complex, emergent systems, where (to my mind) requirements are never completely known from the beginning.

Complaints:

• no reference is made to feature modeling, which seems nearly identical;
• a bad case of EMS (Excessive Math Syndrome): too much detail about cohesion metrics, or algorithms for system design, when the real issue is coming up with the source numbers (anyone remember GIGO?);
• conclusion marred by the confounding factor that the RE-system might just be poorly designed;
• no reference to work on autonomic monitoring and correction (see Yiqiao‘s work).

Full marks:

• acknowledging the issue of change in software systems (although I’m biased)
• actually proposing a theory and testing it!
• discussing threats to validity

What I liked the most about this paper was its fairly scientific setup. I don’t agree that capability engineering is the panacea, but at least I can make concrete objections thanks to their well-structured paper.

Many of the tools I’ve worked on are fairly heavyweight, and assume a fairly large degree of co-operation from the user regarding his/her commitment to the modeling process. But is such a tool important for most users?

Siemens, for example, presents numerous reports each year at the conference arguing for heavy process around requirements engineering, but since it is the RE department presenting the research, it’s hard to tell how much these tools and methodologies are appreciated internally.

Microsoft, apparently, uses documentation extensively in development (security analysis, i18n analysis, performance analysis, etc), but one might argue that this is hardly an endorsement. I’m not familiar with what Google tries, but my sense is that they are much more about the competitive environment for new ideas (and have many fewer challenges with respect to legacy systems support).

Cringely has a good post about organizational support for innovation. In it, he quotes the following: “You wouldn’t need “change management” if you made continuous improvements at the functional level the responsibility of every individual and team cluster (Janna Raye),” So it would seem that in the industrial context, there is a big difference in how, and what, requirements practices are used.

The questions that one must consider is clearly one of future directions. Jorge Aranda‘s paper suggests that for many small and medium-sized companies, at least, the use of requirements models (and associated tooling) will be (is) negligible. The business analysts, who are probably also programmers, generally understand the environment very well, and tweak their product to accommodate new clients, rather than doing revolutionary design.

Greg Wilson characterizes one dimension of software systems according to how agile or stable they are. An agile system to him is one that tries to duck the punches of unanticipated evolutionary changes; a stable system tries to build itself so solidly that these changes cannot affect it.

Martin Fowler, in turn, characterizes these as evolving or planned designs. Clearly for planned designs tools and models become more important (though arguably still not necessary). Evolving designs focus more on working code, code as documentation, and testing as the way to verify the match between requirements and implementation.

A colleague of Fowler’s at Thoughtworks proposes Consumer-driven contracts, essentially user stories for web services: “To recognise the specific benefits and outcomes supported by a service, we need to understand that service in its collaborative context.” How they help with change management: “consumer-driven contracts help contextualise compatibility issues based on extant obligations and relationships. A change to a mandatory element need not always be considered a breaking change; rather, changes that require some form of versioning can be identified with reference to the consumer contracts they break. If existing consumers aren’t using a mandatory message element, then changing or removing that element need no longer be considered a breaking change.”

This turns the burden of change management over to the client, requiring him/her to determine his/her needs, and then managing the change in that context. Previously, a new release from, for example, Oracle, required a fairly binary decision from the IT department: thumbs up or thumbs down on the new product update. In the service world, the promise is that such monolithic upgrades will be a thing of the past.

The requirements problem doesn’t disappear, of course. Indeed, a consumer-driven contract is a form of customer requirement, which should be (largely) satisfied to build a successful project. What is changing is the form these requirements take: rather than large, static, possibly stale models, the requirements might take the form of acceptance tests, behavioural checks, or quality of service policies.

In terms of my research, which focuses on unanticipated changes to a system domain, the relevant questions surround a) understanding how this type of evolution might be best characterized (yes, yet another taxonomy) and b) what type of tool support might help manage these situations. My current thinking is that while change is always unanticipated in some form, we should attempt to support processes for deciding how to adapt to that change.

the core problem, defined

Below are the ways these things are defined. Essentially, we are seeking a set of plans, P, that will satisfy the functional requirements, G, the qualities, Q, according to stakeholder attitudes, A, without violating the domain assumptions, K.

I hadn’t read this paper until it was recommended to me by an advisor, but it is now one of my favourites. It describes an experiment the authors ran on three industrial software projects. They had access to the requirements and design phase meetings from these companies (NTT, Andersen, and MCC), and coded the discussions based on three variables: the target of the question (subject of the question, categorized as evolve, task assignment, interface, realization, and identity), the attribute of the question (who, what, how, why, when), and lifecycle stage (req, design, impl, maintenance).

The results were somewhat surprising. While most questions referred mainly to requirements or design stages (not surprising, since these were planning meetings), few questions sought the intentions behind a decision. Many more questions were oriented to ‘what’ was being changed/affected (~60%), and ‘how’ that was to happen/happening (30%).

Practitioners might not find this surprising; after all, pragmatically those two questions are most relevant to “getting ‘er done”. However, at the time (1993) there was a lot of work on design-rationale tools, which specifically targeted the ‘why’ questions. This is also a central argument behind early-requirements analysis: that understanding intentionality will improve how well a tool performs for the users. Now, I think an important distinction is between answering ‘why’ a particular implementation is necessary, or why the change was made, versus a better understanding of user goals. User goals describe ‘what’ the tool should satisfy, while design-rationale is answering ‘why’ a certain change is being made.

The authors present four possibilities for ‘why’ questions having such low representation:

1. the information is perceived to be unimportant or avoided by people conscious they are not domain experts;
2. these questions are important, but these meetings are not the venue for addressing them, or the context is clear to the participants;
3.  ’what’ and ‘how’ questions provide enough information to elicit or understand the ‘why’ questions;
4. it is hard to answer these questions with current tools;
5. (least likely) while low in frequency, these questions are greater in importance: this suggestion seems unsupported by the data; e.g., the minutes didn’t reflect that such questions are more important.

Another critique that can be made, in a similar vein, is regarding the outcomes of these projects-to-be. I would imagine that studying the design meetings of ultimately failed projects doesn’t tell us much about what ought to be done in these meetings. Perhaps they focused to such a great extent on the implementation details that they omitted other important considerations.  However, the fact the paper has used three fairly distinct datasets strengthens the results considerably.

The paper concludes with an observation that usage scenarios are very important according to the data; this supports the agile focus on use cases and user stories.

All in all an excellent paper, grounded in empirical and externalizable data. The one regret I have is that (to my knowledge) the data is not publicly available. It seems like a rich source of data for a wide variety of questions. While I understand the privacy and IP problems involved, one would think that there is either a way to anonymize it, or that 10 years later, it would be of very little relevance.

ICSM is about software and machine artifacts, not requirements. Requirements come from on high, and the impetus for maintenance tasks is generally assumed to be well understood. It seemed a little like solving the problem of getting suburbanites to a downtown office by improving the highway signage, or improving offramps, but ignoring urban rail as an option altogether.

A discussion at the workshop on evolvability spurred me to consider the ‘smart-monkey syndrome’. My contention is that building software, while hard, is not the hardest problem. I think there is a lot of evidence for this, starting with Brooks’s law, which is all about the external effects on software development, specifically communication.

Along the same lines, Bruce Eckel posted a neat thought-experiment, asking, “If Microsoft, with all the money and smart people it has, can’t release cool applications more than once every few years, than who can?” Several responses disagree, suggesting Microsoft is actually quite innovative. However, one of the respondents posted an argument with which I wholeheartedly concur, namely that it is again the external factors — multi-national lawsuits, legacy support for code from the 80s, internal communication, device support, etc. — that is the real bottleneck. This, I think, supports my contention that the building of software — e.g., GMail — is the easy part. I am almost certain that Microsoft engineers have had many of the same ideas as Google, but external reasons prevented their realization, such as corporate strategy, marketing, etc. Much as I have shied from the business side of software as fuzzy and pseudo-scientific (meaning that many of the claims in the literature are wholly unsubstantiated by evidence), I am coming to realize that it may be the one aspect that matters the most – and certainly more than language choice.

i* models are used for capturing agent-oriented intentional properties in a domain. One purpose of i* modeling is to enumerate alternatives and design opportunities. But the purpose is also to cast new light on the existing state of affairs — who depends on whom, etc. When these two states are combined, it becomes difficult to evaluate a design alternative: are we assuming Situation A will still hold; how can we be sure the Environment will still be the same; when is the shift in mood going to happen. For example, if I’m considering whether to introduce a technical wiki for storing common solutions in a business, does my i* model reflect the existing situation (no wiki) or the desired situation (wiki).  How can I distinguish between the two versions? Often we want to evaluate models to see how well certain decisions achieve strategic goals. How can we do this in a repeatable way if the same model is used for both current and desired conditions?

Another confusion is between structures of the domain and actions in the domain. We often say that a certain task satisfies a particular goal; but we really want to enumerate, initially, these structural properties of the domain — what exists now — and allow the designer to choose the implementation path. The tasks that are possible depend on the objects in the domain and certain basic constraints (time/space, physical laws, etc.). If we try to describe tasks that are necessary, or that exist already, we fall into a trap of descriptive analysis, where what already is confines our new design. Cognitive Work Analysis, for one, insists that structural properties in the domain be enumerated first, followed by tasks and strategies for accomplishing those tasks.

We should have 2 i* models: one enumerating the indicative, structural constraints the work domain imposes; the other providing rationale and alternatives for optative desiderata in the Machine-to-be.