Semantic Werks

An emerging trend in software research is a focus on complex software systems. These systems are typically:

• decentralized: a lack of hierarchical control.
• display emergent behaviour: unexpected behaviour arising from unexpected interaction effects.
• subject to network effects: a rich mixture of human and computer agents.
• large-scale: millions of lines of code, thousands of agents.

An example of such a system is the stock market, or the US military’s future soldier program. Two major initiatives have been developed to focus on these systems: the Software Engineering Institute’s ULSS program and the Large-Scale Complex IT Systems (LSCITS) program in the UK. According to these programs, there is a “Software Problem”, and the tools and techniques we use today must be radically improved if they are to manage these new challenges.

But as Peter Norvig illustrates in this article, we have managed to scale from the half-million instruction programs of the 1970s to the hundred-million instruction programs of today. So why do we expect not to handle the billions of instructions in a ULS system? I’m not suggesting we build software perfectly today. There are many improvements possible. But I’m not sure I agree there is some fundamental challenge we cannot currently address.

One thing I think is important is the ability to fail gracefully. One lesson that seems to have been learned over the years (and the idea surfaces from the start in SE) is that iteration is fundamental. That means building systems such that failure is not fatal. Naturally, in some places that isn’t possible (medical devices perhaps). But even in these seemingly obvious examples, it should be possible to fail usefully: to alert someone to the problem, supporting remote software updates, or incorporating more adaptation capability.

The point of this post is to ask whether the statement: “Our ability to develop, maintain and manage such systems is falling behind the growth in their complexity.” has any evidence behind it. Standish reports need not apply.

I came across a paper[1] via the Nebraska-Lincoln Empirical Software Engineering site that ranks software engineering schools and scholars. The authors assess importance based on citations in IEEE Transactions on Software Engineering, ACM Transactions on Software Engineering and Methodology, and the conferences ICSE, and FSE. I think most researchers would agree these are the top venues for software engineering, although I might quibble with the merits of the journals, since important new research is typically sent to conferences. Also, when a conference has an acceptance rate in the single digits, it tends to drive people to more specialized venues, which aren’t incorporated here, e.g., RE, ASE, Models, and so on.
The list is hidden in a PDF behind the ACM’s “Digital Library”, which isn’t a library at all, since users have to pay a lot of money to access it. So I’ve reproduced it here, because it’s interesting. One thing to note is that UofT, my school, is 26th, and behind UBC, Waterloo, and Carleton. There is a perception at UofT that we are not just the ‘best’ in Canada, but by far the best. Clearly, like Avis, we need to try harder. At least in these four venues.

Top 50 institutions

Rank Institution
1 Massachusetts Institute of Technology
2 Carnegie Mellon University
3 Georgia Institute of Technology
4 University of Maryland, College Park
5 Oregon State University
6 University of California, Irvine
7 University of British Columbia, Canada
8 Politecnico di Milano, Italy
9 University of Texas, Austin
10 IBM Thomas J. Watson Research Center
11 University of Waterloo, Canada
12 University of Massachusetts, Amherst
13 Imperial College London, UK
14 University College London, UK
15 Carleton University, Canada
16 University of Paderborn
17 Purdue University
18 Stanford University
19 Kansas State University
20 Katholieke Universiteit Leuven, Belgium
21 Michigan State University
22 University of Pittsburgh
23 University of Colorado, Boulder
24 University of Texas, Dallas
25 University of Washington, Seattle
26 University of Toronto, Canada
27 Ohio State University
28 University of Southern California
29 University of Karlsruhe, Germany
30 Osaka University, Japan
31 University of California, Davis
32 Fraunhofer-IESE, Germany
33 University of Virginia
34 Simula Research Lab, Norway
35 Washington University in St. Louis
36 Hong Kong Polytechnic University, China
37 Brown University
38 University of Illinois, Urbana-Champaign
39 University of Strathclyde, UK
40 NASA Ames Research Center
41 University of Bologna, Italy
42 University of California, San Diego
43 Avaya Labs Research
44 Northeastern University
45 West Virginia University
46 Case Western Reserve University
47 Rutgers University, New Brunswick/Piscataway
48 Bell Lab, Naperville
49 Institute for Information Technology at National Research Council, Canada
50 National University of Singapore, Singapore

Top 50 scholars

# Scholar
1 Harrold, M.
2 Rothermel, G.
3 Murphy, G.
4 Briand, L.
5 Ernst, M. (not related!)
6 Jackson, D.
7 Kramer, J.
8 Uchitel, S.
9 Mockus, A.
10 Egyed, A.
11 Magee, J.
12 van Lamsweerde, A.
13 El Emam, K.
14 Emmerich, W.
15 Chechik, M.
16 Batory, D.
17 Inverardi, P.
18 Devanbu, P.
19 Herbsleb, J.
20 Clarke, L.
21 Jorgensen, M.
22 Robillard, M.
23 Soffa, M.
24 Sullivan, K.
25 Letier, E.
26 Stirewalt, R.
27 van der Hoek, A.
28 Bertolino, A.
29 Dwyer, M.
30 Krishnamurthi, S.
31 Tonella, P.
32 Basili, V.
33 Kitchenham, B.
34 Taylor, R.
35 Memon, A.
36 Michail, A.
37 Dingel, J.
38 Notkin, D.
39 Walker, R.
40 Orso, A.
41 Roper, M.
42 Griswold, W.
43 Kemmerer, R.
44 Leveson, N.
45 Padberg, F.
46 Roman, G-C.
47 Sinha, S.
48 Tian, J.
49 Engler, D.
50 Elbaum, S.

[1] Jie Ren and Richard N. Taylor. Automatic and Versatile Publications Ranking for Research Institutions and Scholars. Communications of the ACM (CACM), Vol 50, No. 6 (June, 2007). http://doi.acm.org/10.1145/1247001.1247010

Update:

I came across another ranking of researchers. This one [2] is specific to requirements engineering. Here, the University of Toronto fares much better:

[2] Alan Davis and Ann Hickey and Oscar Dieste and Natalia Juristo and Ana Moreno. A Quantitative Assessment of Requirements Engineering Publications — 1963–2006. in proceedings of Requirements Engineering: Foundation for Software Quality, 129–143, 2007. [BibSonomy] URL