I study software, not software engineering
Software research has always been a strange creature. Born in the 60s as computer pioneers realized the immense challenge software posed (where previously most efforts had centered on hardware engineering and design), software covers such a broad scope of domains and possibilities that categorizing it is like categorizing transportation conveyances. Are you building an ocean tanker? A cheap, self-propelled city vehicle? Something to destroy buildings?
That’s why most arguments about software tend to feature wagon-circling by members of particular domains. Are we engineers or craftspeople? The guy who builds websites for local veterinarians is appalled by the demands RUP places on his development process. He finds Java and Tomcat ridiculously over-engineered. At the same time, the multinational IT support person can’t fathom how anyone would develop without a fully elicited requirements model or class diagram. She looks at Ruby and sees a newfangled bastardization of functional programming and Perl.
These aren’t the only camps, either. It often seems to break down over project size, potential harm, and importance. NASA has very few defects in its code, but spends billions of dollars to get to that point. Not something 37 Signals is interested in doing.
This is why I find the term ‘software engineering’ misleading. Sure, back in the 60s, 70s, 80s, venues like ICSE were largely about giant defence projects, things like Star Wars (see here for a great rant about how futile that effort was, though). Software could be engineered, right? Throw peons at the project, a bunch of money, and with a healthy management team the project would succeed.
Since the dawn of the open-source movement, I would say, and the rise of personal software in the late 80s, research on large-scale software was increasingly less useful to a large number of practitioners. This led to a number of non-academics proposing truly innovative and successful methodologies — Scrum, FDD, XP, Lean/Kanban — because, in my view, the researchers just weren’t able to see the change coming. Partly this is because as a professor, you need secure funding, and small firms cannot provide this. Inevitably you end up working on Oracle instead of MySQL, for example.
The exciting things is that doing research in software fascinates me precisely because of this diversity. But that research must be targeted appropriately. Don’t insist on the primacy of UML for every possible project — it just doesn’t make sense. Don’t see statements like “people over process” and assume that process is irrelevant.
I think in order to more fully embrace this awesome scope, to make research (more) relevant, we ought to stop pretending we all do engineering the way they had hoped — although not all of them — at the NATO conference in 1968. We need to more carefully identify the relevance of humans in the software loop. We need to accept the inevitability of change and inconsistency in software and organizations. We need to figure out what’s useful for the ten-person company, as well as the thousand person company. We ought to just call ourselves software scientists, and go from there.
Currently, it seems to me that practitioners of all stripes get their information thusly: first, via colleagues or anecdote; secondly, from websites like Slashdot or Digg; thirdly, from vendors and PR agencies; fourthly, from industrial research firms like Gartner or Forrester; and finally, lastly, from academic research papers. I would really like to move academic research up this list.
Good post; thanks Neil.
“I would really like to move academic research up this list.” –many of us do, but not that many want to drop what they’re doing and study research questions that practitioners actually find relevant.
Jorge
2009 June 6 at 08:07
The fact that “we don’t do engineering the way they had hoped” does not imply that we shouldn’t do it. What we should be able to is to find out the right amount of “engineering” for each project (depending on the size of the project, the criticality of the domain,…)
Jordi Cabot
2009 June 6 at 09:40
@Jordi: I think that’s what I mean .. but this is engineering in the purest sense of design and experiment, find a ‘good enough’ solution to the problem. I wasn’t around in the 60s and 70s, but I imagine, like IBM not imagining that there would be ‘a computer on every desktop’, software experts back then would have found it hard to predict the sheer size and variety of today’s software landscape.
@Jorge: One of the things that I think about is relevance; it doesn’t seem to bother physicists, so why do we care? I wish I could close my eyes to the problem like some in the field do. I think the issue, like the CHASE workshop mentions, is that ultimately this is a human endeavour, and so research needs to reflect that.
Neil
2009 June 6 at 14:04
It is a shame that common sense like “certain things are useful in certain contexts” don’t get twittered, redditted, blogged, digged, slashdotted. Only this extremism of opinions gets any play online, see Joel Spolsky, Jeff Atwood, Uncle Bob, etc.
Common sense isn’t interesting or bloggable. No one wants to hear that XP is a really bad idea if you have very strict requirements, no one wants to hear that gee if you have requirements document already made maybe SCRUM isn’t so useful. No one wants to hear “maybe SCRUM and XP are working for you because you didn’t do anything before”.
No one wants to hear this, thus developers will continue to be assailed by consultants pushing the latest greatest things.
BTW you suck at SCRUM and you should hire me to tell you how you do everything wrong.
anonymous
2009 June 6 at 10:49
I do not think a web designer is a software engineer, since he is using an application, the same way that a painter is not a chemical engineer since he is using a chemical product. I do agree in the term software engineer because you are building up an application, no matter how complex (or simple) this may be. A civil engineer may be designing the next CN Tower or just paving a driveway, that do not means that he can just become an “craftman” for the later task and do not apply his knowledge and discipline in the required amounts. Not all developers are scientifics, and change and inconsistency exists in many engineers fields, not only software!
Manuel
2009 June 10 at 09:59
A follow-up: Tom DeMarco seems to agree that the notion of ‘engineering’ is outdated, in a recent article on IEEE Software. He thinks we should be focusing more on delivering value, where we’ve seen some amazing success, rather than precision and meeting deliverables.
Neil
2009 July 2 at 11:38
@Manuel
Web design is not the same as developing for the Web. I’m not sure what the point of conflating the two is.
Wyatt
2009 November 30 at 14:20
I came to this blog by chance, but found lot of sense in what you are saying. While building and maintaining software, we need to take care of two diagramatically opposite systems, namely the production system and the innovation system.
The production system needs to be predictable, repeatable, measurable, deterministic, hierarchial and low risk. On the other hand, the innovation systems are uncertain, exploratory, judgemental, ambiguous, cross-functional and high risk.
By whatever name we call it, software engineering MUST help us balance these two systems in the most approrpiate way for a given situation.
Prabhakar Karve
2009 December 14 at 03:10
It is a shame that common sense like “certain things are useful in certain contexts” don’t get twittered, redditted, blogged, digged, slashdotted
martin
2009 December 26 at 03:23