I think my conceptual problem is that MDE encodes a specific process in a tool. At least one central selling point is that the models can be independent of the context in which they are used. That is, we have this process for desiging graphical editors in Eclipse (with a hard-coded architectural approach), and we have encoded this into the GMF framework.

Cynefin framework (via Wikipedia)

It isn’t that this can never be useful. But if we turn to Cynefin, we can see why it might be problematic. I would characterized MDE as working well in “complicated” or “simple” domains, where the appropriate response to a problem is to use best practices or ‘good’ practices – exactly what MDE is doing. However, much of the work we do in software development is in the complex, if not chaotic domains, where best practices simply don’t exist. I would characterize my experience with GMF this way. The tool we built had unclear requirements, and we didn’t understand the technological landscape properly. A better approach, if we go by Cynefin, would be to “probe” the problem by building prototypes, and then determine what our response to the experiement would be.

In other words, if you are in a simple or complicated domain, then MDE tools should work great. They will allow you to deploy your app to multiple platforms, greatly reduce re-engineering effort, and much else. But if you don’t understand the domian very well, or you are in a chaotic system, you should put the MDE aside. And it’s my opinion that most software development occurs in the complex domain.

Reasons

At UBC, we have for some time used IBM’s Rational Team Concert tool, which is free for academic use, as our software collaboration environment. This was the default tool for this term, as well, save for the three groups who applied to use GitHub. The University of Victoria has been using RTC for a similar purpose.

RTC is, I’m sure, an excellent product for its intended audience, namely, professional software development teams. It is easy to install and maintain for the technical support staff here, has sufficient documentation, and clients for Mac, Linux and Windows. It is also free for us to use as part of the IBM Academic program. However, in course evaluations, it has been the single most complained-about part of the course. It is cumbersome to install for students, and most importantly, always seems to be out-of-date with other software. In our case, RTC 3 is built using Eclipse 3.4, which is “somewhat” incompatible with the libraries and plugins I was looking to use, chiefly the Google Plugin for Eclipse. A significant amount of course time (TA and instructor office hours) was spent getting RTC to work with the other software for the course. And I am just seeing that IBM is now working on RTC 4, which implies more trouble in the future.

Now, this is partly because students have not had much experience installing commercial development tools on their machines, and that is certainly a learning objective for this course (I am confident it is a pain point for nearly all software developers, new or otherwise). It is also because students run a bewildering array of operating systems and language profiles on their laptops and desktop computers, which makes support a headache.

That being said, my sense was that RTC was simply too much tool for what the students needed. As Greg Wilson’s DrProject experiment showed, students simply do not have time, nor inclination, to leverage the more powerful collaboration aspects. Filling out work items, creating documentation, even committing code is something they just do not see the need for. To get them to try it, we must assign marks to those activities. RTC’s terminology (streams, components, etc) is probably great for a developer with multiple projects: for these students (and me!) it is non-standard with most version control concepts and confusing to use.

Since I’ve personally used GitHub for a while, and git seems to have a lot of developer mindshare, it seemed like a good fit for an experiment.

Experiences

Instructor

I emailed Github about the use of their web app for education and received a very prompt affirmative. Github will provide an organization account to the instructor, which includes private repositories for up to 200 people. At the end of the semester, they then require you to either delete the repositories (on Github, obviously not locally) or make them public (free accounts).

The Github UI is generally simple, but some of the navigation options are confusing from the team manager point of view (me!). Tracking student performance is pretty easy, since you have ready access to the excellent Github website, including issue tracking, change set tracking, pull requests, etc. Github has excellent graphs that allow the instructor team to check who is doing what. There doesn’t appear to be a good way to email all of the students in the various teams at the same time (we are an “organization” made up of 4–5 person teams).

Git itself is the main reason to use Github. There are vastly superior tutorials on it, and I like the pure distributed model better than what RTC provides. Finally, and perhaps most important, as the instructor I’ve used Git a fair bit and RTC very little. The disadvantages are that there is no central repository for backup purposes, although being distributed this is presumably less important.

Student

Students were generally positive. The alternative, in this course, was to use RTC. Git has the advantage of more widespread adoption (you’re unlikely to use RTC at Microsoft, but MSFT supports Git in various places). And of course, if Github goes down, then students can no longer manage issues. Git itself is complex to learn; I should have provided a short tutorial to those teams on the basics of Git.

They also found tools like SourceTree and Tortoise invaluable in understanding what was happening with branches and remotes. For a while, a few teams had multiple, non-merged and conflicting branches for each member, which they could resolve once they saw visually how the branches were happening. The concept of remote repositories and pull requests is a little alien at first.

Issue tracking in Github is primitive relative to RTC. This is a strength for this course, I feel, but when we go from user stories to tasks it means students had to roll their own classification scheme (e.g., define a product backlog item, then the tasks which compose it).

The teams which used Github were much stronger than most other teams in the course, so the results are no doubt skewed. That being said, I don’t think RTC was any simpler to use—in a number of teams, at least one team member never managed to commit code to the shared repository.

Looking ahead

The obvious question is, “Would you use Github again”? The answer is yes, and perhaps even “I would like to make everyone use it.”

It was confusing to have two separate tools in the course. Partly, this is because marking is complicated by the fact student teams are in tutorial sections, so some teams in a given tutorial were using Github and some RTC. This meant TAs had to mark both tools (and learn both). Exam questions are more complicated, since you must account for some students never having used RTC, if you ask about issue tracking.

I like the fact that the project collaboration tool was separate from the IDE. I think RTC’s tight Eclipse integration makes it difficult to install the IDE necessary. Some students ran Eclipse 3.7 in conjunction with RTC (Eclipse 3.4) in order to get plugins working. Since git is so popular, it is much easier to find tool support for it than to munge RTC into your work flow. In future, tools like Mylyn would be useful to better integrate issue tracking into the IDE.

The big outstanding issue is privacy. In BC, the provincial government is considering laws that prohibit (or seem to, IANAL) student data being anywhere near a US server (despite students happily sending email about their marks from Hotmail or Gmail). While I respect that motivation, I feel there should be some way to give consent, particularly when so many excellent tools are US-based.

The critique against management as measurement (e.g., Jorge Aranda‘s) stems from a belief that measurement implies a Taylorist approach to software engineering that is simply not applicable. Software construction is not widget assembly, and even in industrial assembly lines the notion of human workers as cogs has changed dramatically, most obviously as encapsulated by the set of philosophies called the “Toyota Way”, and now commonly called “lean manufacturing”.

The idea that a software team lead could sit in a corner office, enveloped by dashboards showing lines of code added per day, bugs fixed, etc. does seem silly. Clearly, a better approach would be to understand the team dynamics and their struggles – the role that the Scrum Master plays, for example. However, this isn’t to say that metrics aren’t supremely important. The lessons of big data and Google-style machine learning is that there are often messages hidden in the data that cannot be surfaced with individual, anecdotal experience. Consider a simple indicator like “lead time” (the time from a work item being created/accepted to it being closed/released). It can be very difficult to get a good sense for this in the aggregate, since developers spend so much time on individual items. It is only looking back on a cumulative flow diagram that spikes in lead time can be detected, and bottlenecks resolved.

I think the difference from Taylorism is that modern metrics are seen as complementary to hands-on, qualitative management, or pieces of  a software process improvement approach like Kanban. In Kanban, two key areas where this occurs are the principles of “Visualizing workflow”, often using a whiteboard with swim lanes, and “Managing Flow”, which uses lead-time and work in progress measures. The key point, though, is that these are two of the five core principles, the others being “limit work in progress”, “make process policies explicit”, and “improve collaboratively”. You don’t manage on metrics alone 1. The metrics drive the self-improvement process but cannot replace in-depth understanding. In short delivery cycles like Scrum or XP advocate, this is much easier to do, because the ability to tweak the process occurs much more rapidly than in longer-iteration approaches, as Jim Highsmith pointed out.

1. Let us stipulate that there are endless examples of low-maturity teams out there whom no technique will help.

Finding this partition for any task will require research to think of a different way to decompose the task. It will require a thorough understanding of the task and of what is algorithmically possible. It will likely require ingenuity in finding perhaps multiple, orthogonal lexical proxies for the semantics of the task, whose combined capture of false positives is significantly reduced from that of any one lexical proxy. For any task, the partitioning will take into account
– the burden to the human analyst of the imprecision of the clerical part and – the difficulty to the human analyst of the thinking-required part.

Obtaining this information will require research like that done by Dekhtyar et al. [14] for tracing tools to determine what is really difficult for humans and how well hu- mans perform parts of the task with and without automation. Addressing the issue of how to separate the clerical and thinking-required parts of a task is of course one of many research questions that challenge the developers of NLP-based tools for RE. The challenge of ensuring industrial adoption of the tools remains.

This is a great motivation for my immediate research direction. My tool, RE-KOMBINE [2], can automate the process of discovering possible solutions a given set of requirements. It can provide information to the analyst about the optimal set of user stories (or use cases, requirements, what have you) to prioritize for development. But it should not, according to Berry et al., try to automatically generate solution, nor should it concern itself with how the requirements are entered into the knowledge base (indeed, our support of natural language propositional statements seems as much formality as one can demand). Finally, it seems the most important research challenge is “determining what is really difficult for humans” in RE.

For more on RE-KOMBINE you can check out our CAISE 12 paper [2].

[1] Daniel Berry, Ricardo Gacitua, Pete Sawyer and Sri Fatimah Tjong, “The Case for Dumb Requirements Engineering Tools“, REFSQ, Essen, March 2012.

[2] Neil Ernst, Alexander Borgida, John Mylopoulos, Ivan Jureta, “Agile Requirements Evolution via Paraconsistent Reasoning,” CAISE, Gdansk, June 2012.

In software development, many advances (by which I mean “better ways of developing software”, where better is defined with respect to perhaps cost, time, value-delivery …) have come from industry and consultants proposing solutions to the problems they have come across¹, of which some examples include Scrum, Kanban, Go, Continuous Delivery, etc. The argument could be made that one person’s hundreds of consultations with troubled software projects provides him or her much more in the way of actionable observation than does a researcher’s single site visit at a large organization, even if the latter is rigorously conducted. Several researchers have argued that the case study is not necessarily non-generalizable, and that it is probably the most useful empirical method for software engineering. However, I think this raises a bigger question: how does a well-conducted academic case study compare with the plethora of anecdotal evidence from software consultants?

Consider this blog post by Martin Fowler. He is a well-respected member of the software engineering community, a thought-leader if you will, having written important books and posts on refactoring and agile software development. In this post, he talks about the notion of adding members to a software development team, the difficulties with which have long been recognized, starting with Fred Brooks and the IBM os/360 project. Fowler makes a few interesting points about this process, and as evidence for his beliefs, cites some of the many consulting experiences his team has encountered. The question for an academic, then, is how your efforts can be compared with his. Is his experience any less generalizable to theory? Is his data less valid for being encountered as part of a for-profit consulting gig? How can a researcher compete with the level of access Fowler has?

Ultimately what troubles me is this question: is the knowledge which Martin Fowler gains via these visits any less useful to us than that which is collected through a more “scientific” case study approach? While we would certainly reject a paper which did not explain the research questions, data collection approach, and discuss threats to validity, none of which are mentioned in his blogs (and why should they be!) I would argue that Fowler’s experiences are in fact more useful: he is extremely experienced with software development, and the breadth of cases which he encounters makes the problem of research question moot. By at least one standard his knowledge has been directly applied: one can argue that the hugely influential “Refactoring” book is a direct result of his anecdotal experiences².

Yin: analytical generalization to theory

The central argument against the complaint that case studies cannot be generalized (and therefore provide no general knowledge, just context-specific knowledge) is that of analytical generalization. This is a complex way of arguing that what you obtain from a case study is simply a singular datapoint about how phenomena are manifested. As the figure below, from Yin’s book³, shows, the notion is that in any form of scientific inquiry we are interested in the theoretical implications (since we assume that the theory is how we make sense of the world). However, I’ve yet to encounter a similar notion to that of “power” in experiment for case studies. It seems it ought to exist, because a case study which merely confirms what we’ve already suspected is less useful than one which, well-constructed, can give us good evidence against a particular phenomenon.

Flyvberg ⁴ makes a nice point about this: if you pick your case study subjects well, then you don’t need to conduct a number of repeated experiments—-one will be enough. This (to me) is what Kuhn and Popper have in common: the notion that intuitively, it ought to only take one good experiment to invalidate one’s acceptance of a theory (where theory might mean as little as “intuition”). Flyvberg illustrates this by pointing out that what Galileo did in establishing a revised theory of gravity was simply a conceptual experiment about conjoined masses. Since he had chosen his case study subject well, he was able to confirm for himself that Aristotle could not possibly be correct. There was no need, in other words, to continue doing experiments with different materials, since by that point he had demonstrated a counter-example and invalidated the theory. He did not even need to concretely demonstrate this to himself, but only show it must be true – the black swan, in Flyvberg’s words.

For me this illustrates the importance of case study selection. A poor selection will only be able to confirm what we already suspect, and in this must be supplemented with additional inquiry. But a well-chosen set of cases—-for example, looking at bug fixing practices in a large industrial setting, in an open source setting, and in a safety-critical setting—-could be sufficient to invalidate a number of existing assumptions. And for me this is also what distinguishes research from anecdote: research chooses cases so that they can serve this purpose. While the anecdotes of Martin Fowler may do this, it is more likely that they are a series of confirmatory experiences, none of which illustrate much other than the fecklessness of management and the importance of concepts such as frequent delivery, customer interaction, and version control—-none of which are remotely controversial⁵. However, should Fowler find a single case where agility hurt business value, then we might have an interesting experience to report (although I suspect those circumstances would be waved off as extraordinary and non-relevant).

It did not take practitioners long to formulate theories such as agile software development (prefigured in any case by the work of Brooks, Boehm, and even Royce’s initial “waterfall” paper). Indeed, most of Extreme Programming was based on a single experience at the Chrysler Compensation system, hardly the stuff of rigour. I conclude, therefore, that as academics, one ought not to be fearful of the extensive industrial experiences of consultants, but instead apply strategic sampling in order to identify truly meaningful cases. I conclude this section with a nice passage from Flyvberg:

Knowledge at the beginner’s level consists precisely in the reduced formulas that characterize theories, whereas true expertise is based on intimate experience with thousands of individual cases and on the ability to discriminate between situations, with all their nuances of difference, without distilling them into formulas or standard cases.

In this light, then, perhaps we can cast Fowler’s anecdotes: that it is innately difficult for him to distill the true nature of these experiences into simple narrative for the beginner.

What distinguishes case study from anecdote?

There are fairly clear methodological guidelines for conducting formal case studies. To begin with, more (even most) problematically, Fowler does not make data available. Either because of time or NDA, there is no way to confirm any of the claims he makes based on the data⁶. This means we cannot in good conscience reject rival hypotheses, since there is no formal proof. A case study is one way of deriving scientific knowledge (parallel techniques include experimentation and ethnographic studies). Since the case study is well-suited to investigating problems in context, it has been argued this technique is most suitable for software engineering, given the well-documented challenges research in this area has with reproducibility and isolating confounding factors.

The most important aspects of a case study is to begin with a clear research question (either exploratory or confirmatory), correctly identify meaningful units of analysis (e.g., development teams, specific bugs, software components), and follow a clear methodology for data gathering (using purposive sampling to gather as much variability as possible). The academic would argue, I suppose, that without a clear research question, and an underlying theoretical perspective (e.g., there is a rich, tacit knowledge in development teams, and bug databases do not capture this information), one cannot generalize to a theory.

This is perhaps the most important distinction to keep in mind, particularly since case studies are usually dismissed as anecdotal or trivial by doubters (who desperately want software to be math or physics). In experiments, we select a (hopefully random) sample of a population, and test a specific research hypothesis on them, attempting to isolate the factors we believe truly contribute to the effect. For example, we may ask a set of students to model a problem with either UML or ER diagrams, and try to measure which technique is “faster”, “richer”, “more understandable”. Assuming our study has sufficient power (large enough sample to be considered representative), and we have correctly controlled for unknown variables (unlikely, in my opinion) we can use this data to generalize to the wider population: for example, with a 5% chance of being a random result, we see that UML is 20% faster at modelling a problem.

In case studies, by contrast, we are generalizing to a theory. Let us say that we believe that bug repositories do not capture all the information inherent in bug reports⁷. We conduct a study at a single company of 10 bugs, interviewing developers, and tracking the bugs throughout their lifecycle. We find that the information that developers know about that particular bug is not completely accessible using artifacts. This suggests that our theory about bug repositories is on to something: at least, it isn’t rejectable, yet. Such studies are often criticized for focusing on a single domain or company, where the findings may be completely different at a second or third setting. But I think this criticism misses the point. We now know more: that this study should be repeated at other places and in other settings does not invalidate the findings, but merely shows their limits. Instead, the criticism should be directed at the entire field: why do we insist so often on “new” results, rather than “true” results?

Challenges with “grounded theory” in exploratory case studies

I think this issue of anecdote vs. case study extends to grounded theory as a way of sense making and theory generation. At best, grounded theory is merely a systematic way of thinking inductively (from multiple “samples” to a cohesive theory). At worst, it is highly confirmatory: you find the data you would like to see. Tagging something with a grounded theory label is a way of justifying the conclusions you drew, of formalizing what scientists do anyway.

But again, without the detailed notes, I have no way of assessing whether the codes you identified are reasonable. There was a nice article I read that characterized (Andrew Gelman?) this type of work as “story-telling”: we seem to be highly narrative as humans, and so cannot resist spinning a good yarn. This leads to reading into the data effects which are poorly supported. The reality is that in many cases, there is a very weak effect at best; however, we get promoted for significant results, not minor improvements. This leads us to draw conclusions from the data that are not warranted. I worry that the same is true of grounded theory.

The real challenge: reproducibility

The real obstacle, of course, is repeating these studies. In isolation very few results mean anything. They must be replicated and reproduced in order to have more validity. This is true of positivist experiments as well as qualitative case studies. Unfortunately very little effort is spent confirming results. Consider the claim that there is a significant factor of difference between good and poor developers. The evidence for this claim is largely old and poorly reported. What we need to do is conduct more studies to test this theory! Yet in software engineering research, testing existing claims is poo-poohed as not innovative. We favour those who generate theories, even if they make little difference or have no good support behind them.

And here is where I believe we can link the academic and the consultant: verifying claims⁸. In the piece by Martin Fowler I linked to above, he makes the testable claim that “any increase in productivity is likely to be proportional to the square root of the increase in people, so doubling a team will get you roughly a 50% increase in productivity”. Of course this is surrounded by caveats, and yet it is something testable. We ought to be doing this as researchers: in many ways, the consultants and talking heads have generated more than enough theories about software engineering for us academics to begin verifying those claims. Consider:

• “Kanban is an evolutionary system for changing workplaces that often leads to revolution”.
• “Pair programming is more effective than programming as an individual”.
• Test-driven development makes for higher-quality code.
• Continuous delivery is preferred to piecewise delivery.

I had some doubts about publishing this list and making my views known, since being on the editorial board for journals like these helps one’s CV. But then, at some point you have to decide whether the larger issues of open access and relevance are important to you. In any event, these venues are, I’m convinced, making it very difficult for academics like me to have meaningful impact on the practice of software engineering. If it will cost a reader 35$ to read my article, I might as well not bother. Which seems to be what the low impact factors are telling us, anyway. (I’m not sure why I linked to them …)

• Journal of Systems and Software – IF 1.277 – Editor Hans Van Vliet
• Science of Computer Programming – IF 1.282
• Information and Software Technology – IF 1.507 – Editor Claes Wohlin
• International Journal of Human Computer Studies – IF 1.600
• Advances in Engineering Software – IF 1.004
• Applied Computing and Informatics – no impact factor, run by the King Saud University (see Crooked Timber for more on Saudi universities).
• Data & Knowledge Engineering – IF 1.717
• Information and Management – IF 2.627
• Information Systems – IF 1.595
• International Journal of Human Computer Studies – IF 1.600

I am writing to express my concern that the ACM is tacitly supporting the Research Works Act. I have read the blog post [1] of M. Chesnais, the ACM president, and to me it reads much like he is an ostrich putting its head in the sand. This act (and others like SOPA) will have major negative implications for research in the computer science disciplines.

That the ACM is international, or that the President is not American, is immaterial. Elsevier is likewise ‘international’, and that has not stopped it from making major financial efforts to influence this legislation. The reality is that US policy and the US market is so important that it has international significance.

I would like the ACM and its affiliates to publicly disavow the Association of American Publishers by cancelling membership. I would also like a public statement to the effect that the provisions of the RWA are anti-science and anti-progress – in short, contravene (at least) Article 2 of the Constitution, which states the ACM’s purpose is to foster the “open interchange of information”.

This issue is so important that neutrality is not acceptable.

Neil Ernst
Member

[1] http://blog.acm.org/president/?p=67

At the same time, or near enough as makes little difference, my wife gave birth to our second son in Toronto. Since everything went swimmingly, she and the children joined me at our place in Vancouver at the end of October. Needless to say life went from work-oriented to family-and-work-oriented in short order.

So the fall has been a little hectic, as we all get used to the new situation. I have found it challenging to get everything properly scheduled (children have a way of disrupting my careful planning). As with most university instructors who also do research, I have found it hard to balance my teaching responsibilities with my research duties. The most effective thing, so far, is to not even allow oneself to consider teaching tasks for certain periods of time. At any rate, it is taking a little while to transition to a new environment and new research goals.

The key for a post-doctoral fellow is to publish good quality papers to enhance one’s academic CV. Since the positions are typically two to three years long, that means you have at most two or three deadlines per conference to target in your time as a post-doc. That makes it challenging, since (in my experience) each paper is the product of at least one year’s work. You can do the math, but essentially time is of the essence.

Our group here is one focused on the software development task, by and large: how humans write and understand code. This makes it a bit different than what I worked on for my PHD, which ended up fairly theoretical, relying a lot on automated reasoning and formalization. However, the purpose of taking the position at UBC was two-fold. One, I think working with smart people of any persuasion will inevitably rub off on you – you need to work with the best to raise your game. If you aren’t feeling challenged and inferior, at least occasionally, I think you are falling behind. More specifically, my intent was to examine my PHD work in the context of software development.

Research questions

As part of my adjustment period to the new environment, I’ve been reading and thinking at a higher level than I typically get time for. I want to refine where I would like to aim my research career for the next five years. To that end, I’ve been listing some research questions that interest me. I list some below, and intend to post more about them in subsequent weeks. I think my overriding focus is starting to clear up: I’m interested in the intersection of business objectives with the software development process. In particular:

• How do business objectives manifest themselves throughout the software lifecycle?
• How do we reconcile a yearly financial cycle with the different cadences of software projects (or design in general).
• How do developers handle software maintenance?
• What exactly is the theory-building approach to software development?

These are all things I knew nothing about when I arrived for my PhD in 2004:

Languages and Tools

• Python (matplotlib, numpy, networkx)
• Lisp (SBCL and Clozure)
• Git and SVN
• Twitter, Facebook, WordPress
• Mendeley
• Latex + Scrivener + MMD3
• Flash / Flex
• Sqlite
• Ruby/RSpec/Rails/Gems

• decentralized: a lack of hierarchical control.
• display emergent behaviour: unexpected behaviour arising from unexpected interaction effects.
• subject to network effects: a rich mixture of human and computer agents.
• large-scale: millions of lines of code, thousands of agents.

An example of such a system is the stock market, or the US military’s future soldier program. Two major initiatives have been developed to focus on these systems: the Software Engineering Institute’s ULSS program and the Large-Scale Complex IT Systems (LSCITS) program in the UK. According to these programs, there is a “Software Problem”, and the tools and techniques we use today must be radically improved if they are to manage these new challenges.

But as Peter Norvig illustrates in this article, we have managed to scale from the half-million instruction programs of the 1970s to the hundred-million instruction programs of today. So why do we expect not to handle the billions of instructions in a ULS system? I’m not suggesting we build software perfectly today. There are many improvements possible. But I’m not sure I agree there is some fundamental challenge we cannot currently address.

One thing I think is important is the ability to fail gracefully. One lesson that seems to have been learned over the years (and the idea surfaces from the start in SE) is that iteration is fundamental. That means building systems such that failure is not fatal. Naturally, in some places that isn’t possible (medical devices perhaps). But even in these seemingly obvious examples, it should be possible to fail usefully: to alert someone to the problem, supporting remote software updates, or incorporating more adaptation capability.

The point of this post is to ask whether the statement: “Our ability to develop, maintain and manage such systems is falling behind the growth in their complexity.” has any evidence behind it. Standish reports need not apply.