Semantic Werks

I maintain (occasionally) a list of common case studies in the RE literature. These are extended examples or model problems, really, which can be used to compare various formalisms. I have links to the data and academic literature.

A common complaint in reviewing research papers is lack of real-world evaluation. But before we get to the thousands of requirements common in industry, we ought to also verify our approach with frequently used examples from the existing literature. I feel that ‘real-world’ is a substitute for voluminous; but often the problem is not (just) the sheer scale, but the interesting edge cases. For that job, I prefer we evaluate our work on smaller, easily understandable model problems.

The page is maintained at the software group’s web page, here.

This is a bit of a contrarian’s view of new software methodologies. First let’s acknowledge that to date, no methodology has reduced the essential complexity in software development, and there is still no silver bullet. That said, it seems obvious that software development has its flaws, although not as bad as Standish tried to insinuate. In other words, there does seem to be plenty of accidental complexity introduced with software development methodologies. It is increasingly clear that so-called ‘waterfall’ development methodologies are replete with accidental complexity (and in my view, accidental complexity is synonymous with waste).

There have been a number of improvements suggested. Two popular methodologies (paradigms, really) are Scrum and Kanban (where Kanban stands for pull-based processes). Both seem to be moving to widespread adoption in the industry (Scrum is ahead of Kanban in this regard, I think). And now the real problem starts. As Tom Grant mentions, this is when the bad implementations start. Someone attends a conference or reads a blog, thinks, ‘map the value stream! we can do that!’ and introduces it to his manager. He’s persuasive, and it’s Friday so the manager wants to go home, so a pilot is approved. The champion may even have some form of certification. The result is a half-assed implementation with no senior management buy-in. What is in reality a complex and holistic methodology, requiring paradigm-shifting levels of commitment from the company, ends up failing miserably. There’s push-back against the process — people start to ask where the high-level design is, a crisis arises pulling staff off the project, etc. This is not uncommon in any methodology — the original ‘waterfall’ paper didn’t actually say “lock in requirements first, then develop, then test, and never talk to the customer”

One of the main problems behind this is a lack of objective information on the methodology. Issue 1 is that while nearly every report you read about the process is positive (of the ‘we implemented 2 week Scrum sprints and reduced delivery time by 123%’ variety), they are often biased and don’t control for confounding factors. These aren’t meaningful case studies, but rather experience reports. They typically don’t have a central proposition defined in advance, they don’t explore rival explanations, they don’t generalize to a theory, and they fail to address threats to validity. I’d caution against reading too much into these reports.

For example, we can’t ignore the issue of expertise. For instance, Paul Graham argues that the success of his Lisp-based website, ViaWeb, shows that Lisp is as good as any other language for web development. However, we have no good examples of follow-on successes, and I suspect that what really happened was a) a smart LISP programmer releasing b) at a fortuitous moment. In the same way, the success of Toyota and TPS doesn’t prove that this system is better than any other. What it does show is that Toyota have an incredibly disciplined and focused organization. To my mind, the pillars of TPS — statements like ”go-see” and continuous improvement — are so abstract that they can basically be translated as ‘always do your best’. They don’t strike me as radically different than what any organization that is driven and motivated would come up with. So if Toyota took market share from GM, was it due to TPS or really just the case of a smaller, aggressive company taking over a sedentary and moribund established player? The gas crisis in the 70s certainly played a part, too.

The second issue behind the failure of new methodologies is financial. Consultants and experts with have a vested interest in adoption, not success. Scrum trainers optimize to generate new Scrum masters, not successful Scrum implementations. Certifications prove nothing beyond an ability to sit through a two-day session. This effect contributes to the problem that mushy middle adoption often ignores the core importance of a tool or methodology. There is a focus on the artifacts — the whiteboard display, the continuous flow mapping, the calculation of velocity — over the huge challenge of getting organizational backing for the process. Moving to a lean paradigm for software development is company-changing. It doesn’t seem to me like something that can be done halfway.

Context is important. This is something that thought-leaders usually mention in their writing, but is somehow lost on the wider audience. What worked in Microsoft will almost certainly not work exactly the same way in your organization. It might be a difference in product maturity, developer skill, management interest, etc.

Lesson: organizational maturity seems to be the chief determinant. It’s like arguing that what matters in childhood education is whether you use phonics or ‘look-say’ techniques to teach reading. This focus ignores the massive contributing factors of teacher skill and parental involvement. I’d say that the best predictor of success in adopting a ‘new’ methodology or  paradigm is the nature of the organization, and not the specifics of that methodology.

p.s. I never once mentioned ‘engineering’ in this post.

Over on my research group’s wiki, I have created a quick list of possible ‘model problems’ for requirements engineering research. Contributions are welcome.

Reading Robert Yin’s book on case studies has really opened my eyes. To date the most contentious aspect has been his distinction between statistical generalization and analytic generalization. The former is what is done in surveys and experiments: using a sample to make statements about the population. The latter is what he argues is done during a case study. A case study is not a sample, but rather part of an argument.

To properly frame the debate, I think it is important to start with the question of epistemology. That is, what is it we are seeking to do when we do research? In my opinion, it is to motivate the rational beliefs we can hold about a given observable phenomenon in the universe. For example, do requirements evolve? If they evolve, what is being done about it? Etc., etc.

Once we have a list of questions we are interested in — why is this apple hitting my head while the moon is always up in the sky? — we can begin to try and answer them. This is where the scientific method chosen is important. In phenomena whose causes can be controlled for, an experimental approach is useful. It allows us to generalize our results to a wider population. But a lot of phenomena in the world are not easily controllable. In this case, we need a technique that can allow us to still draw some conclusions about the phenomenon of interest. Otherwise, we would just throw up our hands in disgust. Is agile software development better than waterfall? Choosing a few good (representative) cases might well let us make some conclusions.

I think one of the important things to remember is that while case studies seem easy to do — just pick a subject and write up a history of the project — they are in reality much more complex to set up than their experimental cousins. This duality is what has led many to discount their scientific — epistemological — value. Most of what is called ‘case study’ in the literature is really more akin to ‘proof-of-concept’.