I had some doubts about publishing this list and making my views known, since being on the editorial board for journals like these helps one’s CV. But then, at some point you have to decide whether the larger issues of open access and relevance are important to you. In any event, these venues are, I’m convinced, making it very difficult for academics like me to have meaningful impact on the practice of software engineering. If it will cost a reader 35$ to read my article, I might as well not bother. Which seems to be what the low impact factors are telling us, anyway. (I’m not sure why I linked to them …)

• Journal of Systems and Software – IF 1.277 – Editor Hans Van Vliet
• Science of Computer Programming – IF 1.282
• Information and Software Technology – IF 1.507 – Editor Claes Wohlin
• International Journal of Human Computer Studies – IF 1.600
• Advances in Engineering Software – IF 1.004
• Applied Computing and Informatics – no impact factor, run by the King Saud University (see Crooked Timber for more on Saudi universities).
• Data & Knowledge Engineering – IF 1.717
• Information and Management – IF 2.627
• Information Systems – IF 1.595
• International Journal of Human Computer Studies – IF 1.600

I am writing to express my concern that the ACM is tacitly supporting the Research Works Act. I have read the blog post [1] of M. Chesnais, the ACM president, and to me it reads much like he is an ostrich putting its head in the sand. This act (and others like SOPA) will have major negative implications for research in the computer science disciplines.

That the ACM is international, or that the President is not American, is immaterial. Elsevier is likewise ‘international’, and that has not stopped it from making major financial efforts to influence this legislation. The reality is that US policy and the US market is so important that it has international significance.

I would like the ACM and its affiliates to publicly disavow the Association of American Publishers by cancelling membership. I would also like a public statement to the effect that the provisions of the RWA are anti-science and anti-progress – in short, contravene (at least) Article 2 of the Constitution, which states the ACM’s purpose is to foster the “open interchange of information”.

This issue is so important that neutrality is not acceptable.

Neil Ernst
Member

[1] http://blog.acm.org/president/?p=67

At the same time, or near enough as makes little difference, my wife gave birth to our second son in Toronto. Since everything went swimmingly, she and the children joined me at our place in Vancouver at the end of October. Needless to say life went from work-oriented to family-and-work-oriented in short order.

So the fall has been a little hectic, as we all get used to the new situation. I have found it challenging to get everything properly scheduled (children have a way of disrupting my careful planning). As with most university instructors who also do research, I have found it hard to balance my teaching responsibilities with my research duties. The most effective thing, so far, is to not even allow oneself to consider teaching tasks for certain periods of time. At any rate, it is taking a little while to transition to a new environment and new research goals.

The key for a post-doctoral fellow is to publish good quality papers to enhance one’s academic CV. Since the positions are typically two to three years long, that means you have at most two or three deadlines per conference to target in your time as a post-doc. That makes it challenging, since (in my experience) each paper is the product of at least one year’s work. You can do the math, but essentially time is of the essence.

Our group here is one focused on the software development task, by and large: how humans write and understand code. This makes it a bit different than what I worked on for my PHD, which ended up fairly theoretical, relying a lot on automated reasoning and formalization. However, the purpose of taking the position at UBC was two-fold. One, I think working with smart people of any persuasion will inevitably rub off on you – you need to work with the best to raise your game. If you aren’t feeling challenged and inferior, at least occasionally, I think you are falling behind. More specifically, my intent was to examine my PHD work in the context of software development.

Research questions

As part of my adjustment period to the new environment, I’ve been reading and thinking at a higher level than I typically get time for. I want to refine where I would like to aim my research career for the next five years. To that end, I’ve been listing some research questions that interest me. I list some below, and intend to post more about them in subsequent weeks. I think my overriding focus is starting to clear up: I’m interested in the intersection of business objectives with the software development process. In particular:

• How do business objectives manifest themselves throughout the software lifecycle?
• How do we reconcile a yearly financial cycle with the different cadences of software projects (or design in general).
• How do developers handle software maintenance?
• What exactly is the theory-building approach to software development?

These are all things I knew nothing about when I arrived for my PhD in 2004:

Languages and Tools

• Python (matplotlib, numpy, networkx)
• Lisp (SBCL and Clozure)
• Git and SVN
• Twitter, Facebook, WordPress
• Mendeley
• Latex + Scrivener + MMD3
• Flash / Flex
• Sqlite
• Ruby/RSpec/Rails/Gems

• decentralized: a lack of hierarchical control.
• display emergent behaviour: unexpected behaviour arising from unexpected interaction effects.
• subject to network effects: a rich mixture of human and computer agents.
• large-scale: millions of lines of code, thousands of agents.

An example of such a system is the stock market, or the US military’s future soldier program. Two major initiatives have been developed to focus on these systems: the Software Engineering Institute’s ULSS program and the Large-Scale Complex IT Systems (LSCITS) program in the UK. According to these programs, there is a “Software Problem”, and the tools and techniques we use today must be radically improved if they are to manage these new challenges.

But as Peter Norvig illustrates in this article, we have managed to scale from the half-million instruction programs of the 1970s to the hundred-million instruction programs of today. So why do we expect not to handle the billions of instructions in a ULS system? I’m not suggesting we build software perfectly today. There are many improvements possible. But I’m not sure I agree there is some fundamental challenge we cannot currently address.

One thing I think is important is the ability to fail gracefully. One lesson that seems to have been learned over the years (and the idea surfaces from the start in SE) is that iteration is fundamental. That means building systems such that failure is not fatal. Naturally, in some places that isn’t possible (medical devices perhaps). But even in these seemingly obvious examples, it should be possible to fail usefully: to alert someone to the problem, supporting remote software updates, or incorporating more adaptation capability.

The point of this post is to ask whether the statement: “Our ability to develop, maintain and manage such systems is falling behind the growth in their complexity.” has any evidence behind it. Standish reports need not apply.

In SE many of the journals I consider the most prestigious have declined in the past three years [1][2]. Consider:

What exactly do I conclude from this decline? Let’s keep in mind that IF is taking the mean of a non-normal distribution, and that the first destination for new research results is almost certainly not one of these journals. Rather, most of the journals tend to publish longer versions of conference papers, invited issues on special topics, and so on. Here are some stats from my last publication:

Number of citations overall: 31
Number of conference papers cited: 13
Number of journal papers cited: 15
Number of workshop papers cited: 0 (other: 1 thesis, 1 book, 1 ArXiv paper)
Ratio of venues for recent work [3]: 4/5 = 0.8

A decline in IF means that the published papers are seeing fewer people refer to them. Then the question becomes, is that because fewer people are doing SE research, or that fewer people are citing journal papers? I think there are a few observations:

• It is difficult to consider impact in a three year span. That assumes that your journal paper influenced people in the next two years. I’m not sure how many of these journals are read by people immediately. For my work, journal papers tend to be those which have shown to hold up over time, i.e. > 5 years. If I want to keep track of current research I cite papers from conferences. This is particularly true if I look at papers only in this area (since other fields tend to favour journals). My 0.8 figure means that of the citations for that paper, I cited 5 conference papers from 2008-2011 vs. four journal papers (all of which were in other areas than RE). Consider this paper by my colleague Sotirios. It was originally published in September 2010 as a conference paper. It has just now come out (September 2011) in a special issue of REJ. But I doubt I will update my citation strategy to cite his journal paper rather than the conference paper.
• Survey journals (e.g. ACM Computing Surveys) and musings on the future of the field should be excluded. By definition these are broad overview papers, which mean they tend to pop up in plenty of paper introductory sections. But they do not represent ‘research’ as such.
• The other factor is diffusion. It may be that the frustratingly long review times and perceived low impact of these journals has contributed to a negative-feedback cycle. It is natural in an emerging science like software research for the community to seek out other venues and create new journals and conferences. Consider requirements engineering. This is a very diverse field which touches on economics, artificial intelligence, human-computer interaction, sociology and many others. It seems silly to think that a single venue could appeal to all researchers. Many of the papers in CACM, for example, are of very little interest to me – probably less interesting than the papers published in Nature. They are simply too technical and too different than my chosen community. I do not know the numbers, but I suspect that smaller venues like MSR and CHASE, which I perceive as having garnered a lot of momentum, have seen increases in impact.
• And the elephant in the room? The fact that none of these venues makes it easy for people to a) find the papers and b) actually read them. For example, for the longest time just getting a list of recent papers was impossible with any IEEE journals. I subscribed to a RSS feed that always had an error when you tried to link through to the paper. It just isn’t acceptable that publicly-funded research is this difficult to access. Impact factor in the large should be about more than just influence on other researchers. We should consider blog posts, software tools, and especially data repositories.

[1] http://www.cse.chalmers.se/~feldt/advice/isi_listed_se_journals.html
[2] Diomidis Spinellis’s blog post on IF
[3] Number of citations in the past 3 years from journals over the number from conferences.

The Latex support in MMD3 is much simpler than the previous version. Instead of complicated XSLT transforms from the HTML formatted Markdown output, the new approach is to transition from the Markdown directly to Latex. It makes customizing the output much simpler – no more editing XSLT files. In the following, I assume you have a Scrivener document that contains the body of your work (e.g., Introduction, Related Work, Observations, Conclusions). Here’s how to get started:

1. Scrivener still ships with MMD2, so you will need to install MMD3 on your Mac. Fortunately this is straightforward. Go to the download page and download MultiMarkdown-Mac-3.0.1.pkg.zip and MultiMarkdown-Support-Mac-3.0.1.pkg.zip (as of July 2011). The support files will seamlessly integrate MMD3 with Scrivener. I’m not sure how easy it is to revert, however, so be careful  (update: pretty easy – see this note). I do think it is ultimately easier to work with MMD3.

2. Now we need to add custom metadata to Scrivener to add the ancillary files for MMD3. I’ve found the easiest approach to be adding a Meta-Data text document as the first document in your Scrivener project (right-click the top folder in the Binder, Add-> New Text). Now we will tell MMD3 where to find the extra files for our Latex output. Here’s what mine looks like:

Base Header Level: 2
Bibtex: IEEEabrv,../../bibtex/thesis-new
Latex footer: ut-thesis-end
Bibliostyle: plainnat-nourl
Title: My Big Thesis
Author: Neil Alexander Ernst
Latex input: ut-thesis-begin


Order matters here. See Fletcher’s guide on metadata in MMD3.

Base header level is telling MMD3 that we want to create Chapters for each first-level Scrivener folder (I think Base level 1 is “Part”). Bibtex is the location of the bibtex files, relative to where we will run the “latex” or “pdflatex” commands. Latex footer will be inserted at the end of the last piece of your Scrivener file using the Latex command \input{}. There is also Latex header, but as we will see that doesn’t work well. The next command, Bibliostyle, will define the bibliography style for use with Bibtex. Title and Author are obvious, and I finish with Latex input. This is the beginning Latex of my thesis document, including packages, newly defined commands, etc. Now, because MMD3 will turn the metadata entries into variables in Latex, it is important that the input come after the definition of the title and author (otherwise there is an error). This is also why I avoid the use of the Latex header metadata.

Now it is up to you to define what document class etc. to use for your document: MMD3, nicely, will just stick whatever is in input/footer/header into the appropriate place in the Latex file. There are some nice pre-defined input sections Fletcher Penney created, that you can download as well on the Github site.

Now in Scrivener, compile your document using the File->Compile.. and setting “Compile For…” at the bottom to “Multimarkdown->Latex”. Note that it is important to disable conversion of two hyphens to an en-dash, otherwise HTML comments don’t work, and you cannot escape the Latex properly.

Note: to escape Latex, surround the Latex (e.g., tables, math) with HTML comments (<!– –>). The most useful MMD features, for me, are lists, which are just numbers or bullets (see the Markdown syntax guide). Much simpler than the cumbersome \begin{itemize} syntax.

To use citations with MMD3, you can use the [#citename;] or [#citename] syntax for \citet{} or \citep{} Natbib commands, respectively. You can also do MMD footnotes with [^foot1] and [^foot1]:Footnote text. I haven’t used any other advanced features of Markdown. The number one wish I have is for easy \ref \label syntax, but I don’t know how to do it (edit: see the helpful post here). MMD3 automatically creates a \label{} after each section heading, based on the section name with no spaces.

I don’t mind writing raw Latex: emacs+Auctex+refTex makes it pretty painless. But I found, for my 60,000+ word document, that it was much easier to do revision and editing in Scrivener: moving sections around, for example. It also uses Mac native spell-check which is pretty nice (Emacs’s spelling I find clunky and slow).

My files for reference:

• Thesis preamble
• Thesis end section
• Subset of MMD output
• Subset of Latex output

I use Auctex with Emacs.app on my Mac to write papers. It’s the best editor I’ve come across for Latex support.

I was having trouble with one feature, however. If your source file contains an error (e.g., an unescaped underscore), the engine via Auctex will complain when you try to compile. In my case, I always use pdflatex to generate PDF output.

At that point you will get "Latex errors in `Document.tex output'. Use C-c ` to display". If you type C-c ` (TeX-next-error), the idea is that Emacs splits into two windows, with one window showing the error message, and the other showing the location of the problem in the source file. For some reason this wasn’t working on my machine, and I kept getting this strange blank window.

After some digging, I’ve determined the problem to be that pdflatex does not report errors using file-name, only line number. This means Auctex cannot determine the proper source file to open.

Following information on this mailing list post, the following steps should fix it (on a Mac):

1. Edit the file /usr/local/texlive/2010/texmf.cnf. This is where user-specific customizations to the general config file for TexLive live.
2. Add the line file-line-error = t at the end. This forces pdflatex to output the file name for each error.
3. Update TexLive using sudo tlmgr update -all which, in addition to updating all the packages in your installation, will recognize your new config option. Possibly there is a shorter way to do this, but who doesn’t want to update their packages?
4. Done!

Consider the case of climate models. Or smaller open-source projects. Here the developers ‘scratch their own itch’ and the requirements are almost always implicit. What to develop next is driven by an internal process, where the Customer (in the XP sense) is always present by definition, since the developer and the Customer are always the same.

And yet these projects often run into trouble because the requirements are not explicitly modeled, where by ‘modeled’ I mean somehow written down and discussed, prioritized, etc. (ed.: Justify this claim!). Backwards compatibility with previous instances is not maintained, reliability is awful, effort is duplicated, and so on. I think only in projects that are being done the same as last time – cf. Aranda et al, 2007 — is there a hope of totally ignoring requirements explicitly.

Why did we start with the BRUF paradigm in the first place? I think it was because like everything in software development, (and computing itself for that matter) things started with the US Defence Department. And their needs are so different than those of a small software development company like 37 Signals that it’s a totally different animal. You have multiple competing vendors, huge safety considerations, multi-year and multi-billion dollar developments. It would be foolish not to do some up-front design. That’s not to say that iterative, frequent release models can’t work in the Army – I think they could. But a little up front planning could save a lot of time designing things you won’t need.

Because the army gives so many dollars in grants, the prevailing academic attitudes are concerned with defence sized issues. The problem is that these concerns are not shared by other sectors. And so small “a” agile started as a reaction to this top-down design.

I’m not defending the IEEE requirements template, or the amount of documentation for CMMI certification. But I very much believe that some form of explicit requirements modeling is important for projects. Look at Scrum — most people are doing user stories with story points. These are requirements models! Simple, but still explicit, prioritized, costed requirements. And people like Scott Ambler argue that to scale Agile, you need a little more – you need ways to organize the stories, to assign responsibility, to manage the stories.

The reality is that when systems get more complex, for most of us it is impossible to keep track of the scope and scale of the system (unless you are Linus Torvalds, perhaps). In that case, to communicate with the rest of the team you will need something to share — and requirements neatly capture what needs doing. I think the word ‘requirements’ itself has come to mean BRUF requirements, when it properly means the set of new properties of the environment your new machine will bring about (Jacksonian requirements (pdf)). Under this definition requirements becomes a much broader term.

One of the big problems is that we understand things in this industry using anecdote. And so many horrible anecdotes have come from “big requirements up front” projects. But in those cases, any methodology would likely fail if the organization lacked maturity. And conversely, any mature organization could use any methodology it chose and succeed. Those organizations know how to motivate people to get excellent work and know not to change scope midway, not to alter budgets, and so on. I think these factors are ultimately much more important than the particular development methodology one chooses.

However, they are a great place to chat with people in your area who are interested in the same things, and the coffee breaks, at least, are worth a dozen emails.

Here’s a few that I wish I could (have gone / go) to.

• Living with Inconsistency
• CHASE 2.0 – Cooperative Aspects of Software Engineering
• The Future Of Requirements Engineering For Self-Adaptive Systems
• WSRCC – Software Research and Climate Change
• Web2SE – Web 2.0 and Software Engineering
• SEAMS – Software Engineering for Adaptive and Self-Managing Systems

We should re-invent the workshop model so we can *all* attend the workshops we find interesting, no? Stay tuned for more thoughts on the matter.