Interpretation Isn’t a Statistical Process

Most evaluations of topic models rely on the structure of the model itself, using word distributions, coherence scores, or lightweight human tasks like word intrusion (i.e., which word does not belong). These approaches implicitly assume that if the structure is good, interpretation will follow.

What they don’t really capture is the act of interpretation itself. So instead of asking whether topics are “good”, we looked at how people make sense of them. We ran user studies where participants didn’t just rate or label topics, they explained them. Those explanations turned out to be the most revealing part.

What became clear very quickly is that people are not making sense of topics as probability distributions. They’re not integrating all the words in some balanced way. They’re using cognitive shortcuts.

How People Actually Interpret Topics

Across participants, a consistent pattern showed up. People would latch onto one or two words, whatever stood out most, and use those as a starting point. From there, they would construct an interpretation by filling in the gaps, often pulling from prior knowledge or familiar categories. This is much closer to heuristic reasoning than to anything like statistical inference.

flowchart TD
    A[Top words in topic] --> B["Salient word(s) selected"]
    B --> C[Anchor formed]
    C --> D[Adjustment using context and prior knowledge]
    D --> E[Final interpretation]

Interpretation as Judgment Under Uncertainty

Another way to think about this is that interpreting a topic is judgment. Participants are given a small set of signals (a list of words), and from that they infer what the topic might represent. There’s no single correct answer available to them, so they rely on whatever cognitive tools they already have: familiarity, category matching, salience.

This makes topic interpretation subjective, context-dependent, and sensitive to prior knowledge. And importantly, it means that coherence alone, the industry standard for assessing topic quality, doesn’t guarantee usefulness. A topic can look clean and still lead people in very different directions.

This follows from what is already known about human judgment and preference, e.g. as captured in Kahneman’s book “Thinking Fast and Slow”.

A Small Example

Take a topic with the words:

apple, microsoft, google, innovation, startup

From a modeling perspective, this is a straightforward “technology” topic. But in practice, interpretation depends on where someone starts. If “apple” stands out, the topic might be read through the lens of consumer tech. If “startup” anchors the interpretation, it might shift toward entrepreneurship. The same word list supports multiple plausible meanings.

What This Changes

The main takeaway for me is that we’ve been misaligned in how we evaluate topic models. We’ve focused on whether the model is internally coherent, when the real question is whether the interaction between model and user produces useful understanding. Once you look at interpretation as a cognitive process, a few things follow naturally:

• evaluation should be grounded in how people actually reason
• interfaces matter, because they can guide or constrain interpretation
• biases aren’t noise—they’re part of the mechanism

If topic interpretation is shaped by (fallible, subjective, human) heuristics, then improving topic models isn’t just about better algorithms. It’s also about designing systems that work with, rather than against, how people think.

This could mean:

• surfacing multiple possible interpretations
• helping users explore alternative anchors
• designing evaluation methods that capture interpretation

Topic models give us structure, but they don’t give us meaning. Meaning emerges in the interaction between the model and the person trying to use it, and that interaction is shaped by all the quirks of human reasoning.

Topics and the Post-LLM World

Topics are very much a pre-GenAI artifact. They provided a scalable and statistically sound way to summarize documents. Coherence scores provided the appearance of rationality. But nowadays, with context windows expanding, it often makes more sense to put the text into an LLM and get the AI to find topics.

However, this still doesn’t address the problems we identified. There are reasons to think the way LLMs process text is similar to humans: they get confused with too much information¹, or they anchor on irrelevancies, or small changes send them in widely different directions. We even call it an “attention mechanism”!

We should continue to challenge assumptions that underpin simplistic benchmarks like SWE-bench, and be very careful with the problems of dealing with bias at scale that LLMs bring.

Causing TechDebt with GenAI

A truism about software development is that code is a depreciating asset (an idea that has existed since the OS/360 work, from Lehman and others). It follows that reinvestment is needed to maintain the asset, and the more of that asset you have, the more you need to reinvest. Furthermore, you really hope someone on the team understands the dark crevices of the asset, the untouched corners that work with some duct tape and baling wire.

Fixing TD With GenAI

If GenAI can emit thousands of debt-laden tokens, causing TD, it can also help us fix these problems. A long-standing challenge, unsolved, is to retrospectively find sources of TD in a codebase. Attempt to resolve this problem have looked at, inter alia:

• self-admitted TD, places where a developer comments the code with a designation like “FIXME”.
• dependency information and code rules to quantify TD, using tools such as Sonarqube or Codescene or DV8.
• metrics such as LOC or the CK suite to identify complex code.
• refactoring detection and support.

The main challenge is to find the TD problems that developers may not know about, but should care about. It is easy to point out that File XY has a method that is 250 lines long. But chances are the devs know about this already and either don’t care or don’t know how to fix it. It is much harder (but more useful) to identify where the unknown unknowns are in a code base.

The Road Ahead

While I think it is too soon to tell if people will still be needed to write code, I don’t see GenAI eliminating TD as a problem any time soon. In the short term, all that vibe-coded software will need someone to maintain it.³ And while GenAI will help us better understand these codebases, I’m skeptical it will be able to properly perform engineering tradeoff analysis. That is something we are actively researching - contact me if you want to help out.

I’m a big believer in the socially constructed nature of software. Too many software problems that I see are the result of human factors, such as power politics or management priorities. It is rare that purely technical problems are to blame. Thus⁴ I do not see GenAI removing the need for teams of humans to figure out what to build, what qualities it needs to adhere to, and how to keep it working.

Peer review, briefly

I’ll start by saying I am gravely worried about peer review and the current scientific model. Quite apart from questions about whether citizen-funded science is worthwhile (it emphatically is), I think AI is exploding the peer review model that has been in place for many decades. AI is writing or at least speeding up whole papers—someone apparently seriously submitted AI generated papers in the dozens to an AI conference—and is being used (against most policies) in peer review. I’ve personally received two reviews that seemed AI generated, e.g., extensive and wordy, overuse of Markdown formatting, bullets everywhere.

If science becomes AI talking to AI, we might as well close up shop as human researchers. There is no reason for people to pay us to simply run AI tools. Of course, I also think the idea of AI taking over entirely in the research process is farcical and usually stated by people who have never done the messy side of science.

Back to the article

Back to Dr Zeller’s article. I agree with most of what he suggests, but I will just point out that the big problem underpinning all of this is the scalability of the humans with integrity who run the whole thing (none of whom, it should be repeated, make much from their efforts).

For example, paper assignments. In Dr Zeller’s model, we ought to stop allowing so many bids, or use of systems like TPMS that are gameable. But what is the alternative? If we receive 1000 papers, who is going to assign that to the 200+ peer reviewers to do the work? How to scale this, because manual assignment is impossible at that scale? And if we have area chairs and editors, how do we know they can be trusted? In the old days of in person PC meetings, we could build trust face to face, and the number of papers was manageable by 1 or 2 moderators. Nowadays the top conferences cannot come close to handling this manually.

Verifying conflicts or checking for suspicious patterns: again, great idea, but who will be asked to dedicate their time to do that work? One would think that is precisely what journal publishers and conference sponsors should be doing, but they are among the worst at trying to cut corners and AI-all-the-things. I can tell you emphatically that being a PC chair is a lot of work, and a lot of dealing with edge cases: papers with AI writing, late reviewers, personal disputes, etc. And while I see being a PC chair or editor as part of my job, and a privilege, one has to wonder exactly how many hours should be dedicated to helping other people publish papers. Certainly a number of community members do far less than their share as compared to papers published.

A modest proposal

My modest proposal¹ is to dispense entirely with peer review, or perhaps have a first hurdle to get into a peer review phase. If a paper gets three people who agree it should be rejected, or weakly rejected, that paper should not have been sent out for review in the first place. One of the most annoying things about AI generated slop papers is the sheer amount of other people’s time they consume.

The obvious response to being asked to work for free on someone’s paper that has minimal effort behind it is to cut corners or simply stop accepting review requests. This is why I also think Dr Zeller’s last point, about incentives, should be more prominent. Collusion and bad behavior exists because getting a paper accepted at ICSE or ICLR or ICML can be extremely financially lucrative, in terms of improved career prospects and state-funded reward structures. If we removed these incentives, or created better aligned ones, that would mitigate against a lot of the bad behavior. For example, two of my colleagues have been ICSE PC chairs, I’ve served on the PC. From what I can tell my employer gives us no credit for these activities in salary review. There are hidden rewards like good vibes, of course, but also recognition(?) and service awards. However, the real focus is on papers published, grants obtained, students graduated, and courses taught. If that is the incentive model (and I doubt it is unique) how can the community move forward?

My Journey

I worked in spatial analysis in undergrad: as an intern, mapping water rights, goat habitat, producing maps for the coast guard. I then worked in my masters with ontologies for cancer biology. In my time at the SEI, I was lucky to work on early planning for the US research software sustainment program from the NSF, which introduced me to the US movement for RSEs, with people like Dan Katz and Jeff Carver.

Since joining the university as a professor, I have been intrigued by the challenges of devloping complex software, in particular, how design choices influence subsequent problems, i.e., technical debt. This also aligns with a wider sense I have that software for climate modeling is a key capability for our future.

Early Work

The earliest work in RSE is basically the field of numerical analysis and HPC. In some sense, the entire discipline of software engineering is derived from scientific software, using simulations to model nuclear weapons, do weather forecasting, etc.

The field of RSE seems to have started between 2000-2005 or so. I think this coincides with general availability of software and the internet and web to connect people. There was a lot of focus on e.g. computational workflows at the time from Carole Goble. People were working on these issues before (e.g., at large US national nuclear labs such as Los Alamos) but there wasn’t a clear definition of the job area.

Jeff Carver’s first workshop is around 2008.

(Segal 2009) is an early paper at CSCW looking at the socio-cultural dimension. A few interesting things:

• One, the idea that the main issue is to support the science. This is the key requirement.
• There is an early focus in some of these papers on the scientist as end-user programmer. This was a popular notion in the 2000s that I think was on the one hand an utter failure (e.g. VBA), but on the other hand, just natural with the right UI (e.g., Excel formulas). And AI will make this much easier. Greg Wilson said something about how when he was teaching scientists in the 2000s, he struggled to motivate why they should not use Excel. Diane Kelly pushes back on this - of which more below.
• A really detailed ethnography on a single lab management project.
• As with most software projects, this one struggled with team dynamics and power.

A followup is (Segal and Morris 2012). This paper is similar to the preceding one, except for a more generic focus on the main differences with conventional development, and the ubiquitous (for the time) focus on Agile.

I should digress here for a moment and say I don’t find software process very interesting. The models - Scrum, Lean, Waterfall, etc. are all highly idealized and in my experience rarely followed in practice. Asking if we should be “agile” is answering the wrong question. You want to deliver things faster and with higher quality, so looking at practices to help that is the key. Anyhoo.

James Herbsleb did some work as well e.g. (Howison and Herbsleb 2013). They looked at incentives.

What Makes Scientific Software Developers Different?

Scientific software has seems to have a different context than a lot of other code. For example, it might be continually maintained. It has different testing needs. RSEs therefore have different approaches. Side note: I am not yet convinced scientific software really is different. A lot of the issues–complex domain requirements, performance intensive problems, team composition–are common in other domains as well.

(Pinto, Wiese, and Dias 2018) and (Wiese, Polato, and Pinto 2020) both report on a survey replication on scientific software developers. Nothing jumped out at me - the problems seem mostly similar to normal development; library problems, stable requirements, etc. Surprising to me was that only 5% of the issues seem science related. But I wonder if that is because the questions were not clear. If one asked where the major cost and development effort is spent, or talked to end users …

(Cosden, McHenry, and Katz 2023) is a survey of RSEs that looks at how they get into the field. Most are domain experts (75%) and the rest are CS grads. Both have different education challenges.

(Carver et al. 2022) continued this; again the findings are mostly interesting as a catalog of the state of things.

A lot of the differences, from what I can tell, is that for a long time RSE was not a career path, and so those folks were temporary, poorly paid, and not recognized.

Testing Scientific Software

(Carver et al. 2007) did a series of case studies looking at how scientific code was maintained, and why Fortran was so popular. I think this paper may have harmed the field, by downplaying some of the complexity involved. It comes across as “this is pretty easy stuff”. But some big projects in the DOD space are studied. Around the same time came (Basili et al. 2008). An observation of that paper is that

• although HPC is focused on “performance” for scientists the performance of the code is less interesting than “time to result”, i.e., a publishable outcome. That time spans writing the code, testing it, running the simulation, etc.
  
• Validation is tricky; outputs are often non-deterministic and probabilistic, inherent in simulating and modelling complex phenomena.
• Programs are long-lived so there is deep scepticism of new tools as plenty of tools cannot make it 30 years. Funders authorizing Voyager, SKA, CERN’s LHC expect the billions of dollars to be used for decades. The software should be able to match that.
• Programmers love being close to the metal, to keep things speedy.

A related paper is (Hook and Kelly 2009), which, while focused on mutation testing, has a nice figure showing the ways error can make its way into scientific code.

TODO: (Babuska and Oden 2004) TODO: (Eisty and Carver 2022)

Scientific Software in Canada

The state of the practice for RSEs in Canada is pretty dire. From a government perspective, we spent a lot of time (and $$) on building infrastructure. That was connecting things with high speed networks (CANARIE) and large compute clusters (Compute Canada). Then, for murky political reasons, there was some transition from those orgs to a central one (The Digital Research Alliance). Unfortunately it seems while the tangible cluster and network stuff continues to get buy in from the main funders, Innovation, Science, and Economic Development Canada¹, the software piece is harder to motivate.

Canada has no research software engineering alliance, like the UK, Germany and the US do. We have no real research labs, like the US DOE labs, and we don’t really do defence research outside of the DND Research groups. We once had software in the National Research Council, but that was axed, again, for reasons I don’t understand but had something to do with cost cutting.

Fortunately, there are some excellent folks in the space who are trying to keep things afloat, a few folks at the Alliance, and some (like me) academics. There are also top notch specialists running the clusters and software support teams at the universities, like UVic’s research computing team.

Things I’d like to know more about

1. how much time does a developer spend on the “science” part of the code, and how much on ancillary roles
2. Can we separate the science logic from the non-science logic?
   1. What is the TD inherent/possible in translating from science to software? Pub pressure, student knowledge, legacy code
   2. “Can we quantify or explain this loss/difference, and articulate the trade-offs resulting from translation?”
3. how do we compare different scientific approaches simply from software alone?
4. how do you retract/code review the scientific code?
   1. what is the equivalent to peer review of the code?
   2. what if the code is a complex model that is unexplainable? how do we test it? where is the science?
5. Can we trace the way in which the design of the code has changed from its initial design to the proper current design?
6. Social debt: how do we check what implications are? How does large team science play a role?
7. Ciera Jaspan’s paper (Jaspan and Green 2023): tools can tell you the current indicators. But what matters is how context defines this as a problem or not. E.g., migrating to Python 3, undocumented Navier-Stokes code. How do we extract this contextual knowledge from a project?

To Read

• Understanding the landscape of scientific software used on high-performance computing platforms
• The dividends of investing in computational software design: A case study

Initiatives

• Better Scientific Software - training materials for RSEs.
• Code Refinery - more training
• Software Carpentry
• Software Sustainability Institute
• US-RSI
• NumFocus grants
• Chan/Zuckerberg grants
• Exascale Computing - Interoperable Design of Extreme-scale Application Software (IDEAS) DOE 5 year software program
• NSF large instrument group
• RESA
• IRIS-HEP
• Collegville workshops

Various “scientific software community of practice” as mentioned in the Connoly article, at UW, CMU, etc.

Venues

• Conferences, meetings, workshops
  • SE4Science workshop
  • Supercomputing conference workshops
  • US-RSE conference - October
  • UK-RSE conference - September. Why these two are so close in time is a puzzle.
  • Alliance Canada Research Software conf. Now discontinued :(
• Journals
  • JOSS (and unnamed proprietary journal ending in X)
  • Geoscientific Model Development (GMD)
  • Computing in Science & Engineering

Meta-research

Here are some papers that have looked at discipline-specific research software:

Example Projects

Glossary

• RSE: Research Software Engineer
• SSI: Software Sustainability Institute
• HPC: high performance computing, e.g., ‘supercomputers’

Metrics

We could look, like I’m sure Apple does, at operational metrics and efficiencies. For example:

• Number of papers in draft/being reviewed/published (WIP)
• Time between idea and paper publication (lead time)
• Students graduated in expected time
• Number of important unanswered emails in Inbox
• To the nearest thousand, how much money is in various accounts, and what the projected “burn rate” is for those accounts.
• How long has each student been in the program, what milestones have they finished, and when they should graduate
• Grant money received
• Grants used efficiently
• Reviews conducted within deadline
• Talks invited/given/follow up
• Size of industry collaboration address book
• Travel reimbursements received within 30 days of trip
• Time between equipment being needed and equipment being purchased and equipment delivered

Now for some of these you might say “but someone else is blocking that!” Which is of course true of EVERYTHING and also not an excuse Steve Jobs was likely to accept. That’s all part of being efficient and operating smoothly. If you know the university takes forever to process room bookings, you need to factor that in to the operational goals.

Why Care About Operations

I think operational efficiency is what separates average researchers from those seen as impactful. Sure, in some cases it might be a brilliant one-off paper, but often we reward output volume: “quantity has its own quality”. Did the project lead to a single paper, or did you harvest 3-4 papers from it? That’s an operational detail that has to do with a PI’s ability to direct students, target appropriate venues, manage meetings to keep the papers on schedule, and so on.

I think the importance of the COO view of one’s career is that for better or worse these outputs are the easiest to turn into data, and subsequently evaluate you, and your institution. So ignoring number of students graduated, or number of publications, or grant values, will result in poor scores on these data metrics. It doesn’t matter how many brilliant ideas you have if no one gets to read them.

The question for this COO view of a research career is to figure out which metrics one truly cares about, and when to stop focusing on operations and think more about strategy and trajectory. Metrics, because the metrics you choose reflect your priorities (e.g., papers published vs industry collaborations nurtured), and strategy, because (hopefully) the research you pursue should reflect some higher level of understanding about what problems are important to be spending time on.

My approach

For me personally, it can be hard to remember to manage the operational details. The easiest way for me to see this concretely is when papers fail to meet a venue deadline. That’s an operational failure: we didn’t move fast enough on data analysis, the meetings were not productive and the project spun its wheels, I didn’t kill the project or value the cost of delay, I answered emails about committee work rather than spending 2 hrs editing.

My current management approach is to check in on each project (I have about 9-10 in various stages) weekly, using a dedicated card (using a note in Apple’s Notes tool). A Kanban board with stickies can be really helpful here too, but the important thing is not the particular system but that you use it and check it regularly.

Another idea I have just started to implement is reflecting on lessons learned from a project (e.g., after a paper is published). Not just the research problems, but the operational challenges. What would I do differently for project management? What worked well in moving the project along? Was this a productive collaboration? Why did it get delayed (it’s always delayed)?

Making Models Accessible

There are two dimensions (or more) to this question. The first is about using models and building them; the second is about querying them.

Model Usability

We now see tons of tools that help with modelling. Here are some illustrative examples

• Fortran, in Global Climate models
• Python, in Ralph Evin’s building sims
• Astropy
• Commercial: Tableau, RStudio, Metabase, https://core.hash.ai/@hash/wildfires-regrowth/9.8.0 are moving from building dashboards and visualizations into more complex modelling support. But like dashboards, the challenge is less about using bars vs lines, and more about the A/B and E parts: what problems do I need to learn about.

One challenge in the move to more complex models is of course the inflection points: the places where the models fail to capture the real world. It is trivial to build a predator-prey model; to build one that accurately captures the dynamics of wolf/elk dynamics in the Mackenzie valley is entirely different, and probably the validity of the model is only understandable by maybe 100 people in the world. Increasingly the challenge in peer review is not about (or not just about) the problem’s relevance or significance, but whether the data and model support the claim. Technical debt in science is a massive concern if we are going to rely on large, difficult to verify datasets for our claims.

How easy it is to build the model:

• Show examples of RStan, Fortran, Hash as dealing with the problem of “building the model”

Accessing Model Outputs: Digital Twins

Bryan Lawrence blog post

The idea that in SKA you might not get raw data

Finding Bugs in Notebooks

Consider our study of how people find notebook problems. Given a notebook on a topic the subject understood well, we asked them to find any problems (0-4) in the notebook. This is essentially peer review of the notebook, except the authors can’t reply or help, like in code review. With “many eyeballs all bugs are shallow” is a flawed truism that seems to hold for science; if we can get lots of people looking we can find some of the flaws.

Hands up if you’ve had to rewrite analysis or model code because of a flawed assumption, misunderstood stats package, or unreliable initialization?

The state of bots

There have been a number of categorizations of bots for software development. The main categories seem to be the ones that Erlenhov came up with, which look at bots as either API endpoints (CI tools), developer assistants, or something more sophisticated. I don’t think there is much to say about CI tooling. This is a spectrum in my view; we will likely see CI tasks grow more sophisticated and multistep.

The paper Peggy and Alexei wrote in 2015 seems to indicate nothing has changed much since that paper. In one sense we are still just doing the same thing; bots as API interaction points, where they automatically or under prompting carry out some well defined and usually pre-existing task like a compiler and compiler flags.

This is in contrast to the bots that appear on corporate sites to maximize engagement and increase sales. There the bot acts as a query refinement tool, helping you to sort out what you actually are looking for. This type of extensive interaction seems to contradict what developers want.

So one question is instead of what bots should do, what tasks are we looking to help with. Bots are useful because they encapsulate common tasks that otherwise humans would have to do. For example we once took punch cards to an operator to enter into the computer; now that is done by hitting compile/run. I had a hierarchy of problems we can get help on:

1. Syntax problems - compilers, integrated into most IDEs, now flag obvious problems before you need to run the code. IntelliJ for example can automatically detect type problems.
2. Warnings and flags - running a compiler with default options just gets the bare minimum. Standards like MISRA-C specify known problems the compiler can find for you.
3. Linters - plenty of problems with code are known, so we can find these things that clearly violate best practices, such as equality checks in Javascript. Often these are integrated into tools like SonarQube or CI environments.
4. Code smells - the next class of code issues are called smells and have to do with slightly more complex problems, often spanning multiple modules. So for example long methods, long parameter lists, and so on. This is also where we might find violations of language paradigms, such as not using list comprehensions in Python.
5. Design problems - the highest level of problems we might ask for help with are design flaws. Here we want to flag issues that we think will impinge on quality attribute satisfaction. We might identify that the code misuses the Obsever pattern.

There are some others that are a bit tangential to these: flagging security problems; identifying dependency or build issues; UI problems; test coverage issues.

The fundamental issue in my view for AI assistance is that it is super easy to tell people what they already know. So in self-driving the issue is not to drive from A to B in the sun; I could probably train my son to do that. What we need is meaningful assistance, telling us the things we didn’t know, or couldn’t know, on our own. Thus bug localization that finds bugs we already know about, code smell detectors we don’t care about, and most other data studies published on effort estimation and so on.

The challenge is that getting tools to the 70% accuracy level is super easy, with today’s tools. But humans are insanely smart, and so that first 70% is not the hard part. It is in the last 30% that we need the help, but also the hardest to decipher; figuring out what the image is really showing in the dark.

I think bots are similar. Right now they are basically just dumb endpoints to an API with a slightly improved interface. Thus Dependabot telling us that a library is outdated is not particularly interesting, since it is just an interface to a more complex script running in the background. What’s novel in Dependabot is that it is able to interact in a well understood way, not the complexity of what it is trying to do. Similarly the bots one sees on airline sites are not interesting assistants, but only simplistic interaction techniques in a web search world. After all John Mylopoulos was working on natural language interfaces to databases 40 years ago.

So what does this more interesting bot look like then? Is it more than just an API endpoint or something else? In my view the next step is truly an assistant that is contextualized to the person asking the question, to the project in which it runs, and the particular time it is being run. It is in short a very efficient, Ms Moneypenny like administrative assistant, capable of anticipating the needs of the developer, but unobtrusively.

How do we get there? There’s a number of pieces for this vision.

• Interface: how do developers like to get information? Right now that is things like IDEs, compiler warnings, interactions on pull requests: working with the artifacts they already care about
• Persona: how should an assistant interrupt? Should they be factual and say we detected a problem on line 50? Or should we omit GUilfoyle’s annoying verbal tics?
• Context: how does the assistant extract the context information it will need to be useful and not annoying?
• Metrics: what would be a relevant way of assessing success? I don’t think we even have a good idea on this.

What Worked

Despite lacking support, getting the contest going on Kaggle was fairly simple. Nicole organized labeling with the rest of DysDoc’s organizers, and hosted a gold set on Github. I then used the gold set to generate the inputs Kaggle wanted. This includes the gold set, split into training and test. Test data is further split into “public leaderboard” and “private leaderboard”, since competitors can submit multiple entries, and see where they stand on the leaderboard. Only the organizers get to see the private leaderboard, which is what ultimately ranks the competitors. You can see that in practice here.

I also had to choose between Kaggle’s available validation metrics, in this case choosing F1, and this can be a bit finicky, as you have to map between the columns in the solution CSV file and whatever Kaggle’s automation expects. Clearly at paid support levels they would make this simpler, or just do it for you.

Despite not having a lot of labeled data, we managed to get a good set of data, and Kaggle’s infrastructure worked - as far as I know, anyway! - with no problems. Competitors download the data, run their model, and then upload a solution file with their predicted labels.

We managed to get 2 principal competitors, one of whom submitted several distinct entries. Both entrants published their submissions at our workshop, which can be found at the ICSME proceedings site.

Improvements and Questions

I was quite happy with how simple Kaggle made the process of evaluating entries. It also scales flawlessly (unlike me), and in theory, could help us dramatically expand our contest. In the COVID era, of course, it also made it pretty easy to host a remote contest, unlike our previous approach, which used in-person demos.

It would be nice to have more support for notebooks, or perhaps a mandatory notebook submission, so that we can see how each group approaches the problem (after it finishes of course).

As far as I am aware, this is one of the first challenges to be hosted on Kaggle. To me it seems like an obvious choice for running and hosting automated benchmarks, such as the various effort estimation and defect prediction datasets out there. If we could disambiguate entries, that would help with understanding who is entering.

Kaggle makes it possible to host an ongoing, never-ending contest, which is also appealing. The obvious bottleneck, unsurprisingly, is data annotation, and at this point I would say that is the main obstacle to running more such contests. However, we have tentative plans to continue the approach in future workshops.

Understand why you are interested in Canada

In the Weimer/Le Goues job seeker’s guide, they make the point that US candidates tend not to move to Canada. I think it is safe to say that if you have spent your entire master’s/PhD in the States, you have worked in the NSF/DARPA/DOD model of funding, and have family ties there, then the switch to Canada would be a big change. I think this is especially true for smaller schools like ours. We’re a small place but proud (both Canada and Victoria!). So make it clear in your cover letter or interview why you would come. Hopefully reading this guide will help!

Explain/communicate the proposed Discovery grant you would win

In Canada funding is fairly different than the US. For one thing, Canadian schools have substantially lower tuition for grad students (although that is changing). Faculty research budgets have much lower student stipends as a result, and the grant sizes reflect that. A moderate US grant might be 100k/year; in Canada that would be equivalent to 20k, but support the same research program.

Your application and interviews should demonstrate you understand this. I suggest reading up at the NSERC page, and also the research services page for the university you apply to.

The main grant for new faculty is the Discovery grant. It is a five year grant worth from $20k-50k a year. You are evaluated equally on your ability/experience with highly qualified personnel; your personal ability as a researcher (i.e. CV); and the research proposal. Not holding a Discovery grant is a problem because getting other federal funding depends on this, to some extent. The good news, especially for ECR, is that success rates are relatively high (60-75%). You can expect to prepare this the summer you get hired, for submission by Nov 1.

Your job talk and your research statement should outline some elements of the five page grant proposal you would write. Departments want to see what you would propose, and how able you are to communicate your vision to external readers. It is a 5 year program, so scope your “future work” to that time frame.

I think this is broader advice than Canada-only, but one thing I’ve noticed is that applicants who are just finishing a PhD give more narrowly focused talks. Two things to keep in mind if this describes you. 1. You will be competing with people who have 2-4 years of post-doctoral training, and a corresponding breadth of research, more experience training students. Stretch your talk to show how you have the potential to succeed like that. Conversely, one question about post-docs is often “how independent can they be”. This is particularly true if you come from a big lab with a famous PI. 2. Think like a professor. What grant areas will you target? How would you manage 5 masters/phd students? How will you balance teaching load with research? I don’t think you need to feel uncompetitive: we invited you for a reason. But the onsite interview is when we want to see if you are ready for what can be is a very demanding job.

Funding

Engagement with industry The federal government has been a big supporter of industry partnerships recently, although the programs were recently overhauled. This typically means that if you have an industry partner with skin in the game, i.e. financial assistance, you have an excellent chance of obtaining government matching funds. Conversely, if you prefer pure research with no immediate outcomes, finding funds might be more difficult. There are very few large granting agencies. There is no equivalent to DARPA/IARPA, DHS, DOD, DOE funding in Canada; those projects would work with specific people at specific agencies to secure one-off funding. In BC, nearly all grants would come via NSERC programs, or MITACS matching. There are also Networks Centres of Excellence such as MEOPAR that allocate funds in targeted areas (these are being phased out). There is also a recent Defence initiative, IDEaS, to increase Canadian funding for research with defence applications. Finally, there were industry-led superclusters announced, but who/what gets funding is still very unclear. It seems to focus mostly on subsidies for industry-led research.

In general, I would say finding funding is much more individualized and distributed than in the States. There are plenty of places to find adequate funding (again, a student probably only costs 20-25k a year), but how to get it is much less clear than a DARPA BAA program. A cynic might say this is because funding announcements are more closely tied to electioneering.

Summer students and internships. We have a similar program to the REU approach, called USRAs. These are government matching for student research semesters. Again, these are allocated on a per-institution basis (bigger places get more).

We have an excellent grid/HPC/cloud computing infrastructure, ComputeCanada. They conduct yearly resource allocation competitions. I don’t know what the success rates are.

For large infrastructure, e.g., robots, 3d printers, tabletop displays, quantum computers, the Canadian Foundation for Innovation holds annual competitions for this, but success rates are fairly low.

Tenure

Well, more to come from me on this one, but my general sense is that tenure is more collaborative and mentoring than many US places. I don’t think there is the equivalent of “didn’t get the NSF grant, didn’t get tenure”, or “didn’t get 1 million in funding, didn’t get tenure”. That said, standards are just as high as US Tier 1 schools; we just want to help you achieve them. We’re friendly, eh?

Specialty hiring

CRCs. You may be in the enviable position of applying for a Canada Research Chair. These are a nationwide funding mechanism for research positions. We have Tier 1 (7 years, renewable, senior) and Tier 2 (5 years, renewable, junior/emerging). They typically come with higher salary and teaching relief. Each university gets a quota from the federal government. The approval process is a bit more involved. In addition to approval from (department-faculty/dean-VP academic/provost), you will have your application submitted to the federal government, wherein the case will be made that you are uniquely qualified, amazing, etc. This is almost never turned down, from what I can tell, but could be. In particular, the federal government has a strong desire to see equal allocations of these CRCs to male and female candidates.

Requirement for hiring Canadians

Departments are usually required to prefer Canadians over non-Canadians, for immigration purposes. This means that of two totally equivalent candidates, the Canadian citizen or permanent resident would be made an offer. If you are a PR/citizen, or applying for PR, that is worth highlighting somewhere.

Immigration is easy

I can’t speak from experience, but my understanding is that immigration to Canada as a permanent resident, and eventual citizenship, is much easier than the US process (with which I do have experience). This is also true for immediate family (spouse/children). In some cases, permanent residency is possible in months, not years.

Salary and benefits

In general Canada pays less salary. Keep in mind that it is a 1 year salary, not 9 months. Most Canadian schools don’t have the concept of a summer salary. At UVic, we operate on 3 equal semesters, and allocate a research semester where you would like (subject to teaching needs of course).

The CRA survey has more useful information. Health care is provincially funded from your taxes, so don’t expect to lose 500-600$ a month to health premiums. From working in the US, even being a well-paid employee at a great employer, there was a significant cost (mental and financial) in understanding yearly plan changes—even without chronic conditions.

In most places, faculty are unionized or quasi-unionized. This means you fall into a grid, and your salary increases will be based on a formula in the collective agreement. You can probably look this up online for each institution you visit. Hint: you want to move up the grid as much as possible before you start the job. So Prof. Le Goues’s advice on startup over salary might change, since your salary will be the baseline for future percentage increases.

Summary

I would sum up by saying Canada is an awesome place to do research, and I hope you apply to Canadian universities! Especially mine!

Resources

• Maclean’s Guide to Canadian Universities: This is the Canadian equivalent (in all respects, good and bad) to US News and World Report. It divides universities into medical/doctoral, comprehensive, and primarily undergrad. Canada does not have the same diversity of higher education as the US—for example, there are few private institutions here. The main division for research is whether the school has a medical school or not, as med schools are tightly controlled (public health care dictates number of seats), and med schools tend to accumulate massive amounts of research funding. My school is categorized as a comprehensive, but I wouldn’t say this equates to “more teaching”.
• NSERC: The main engineering funding body, similar to NSF.
• Taulbee Survey: Various stats on academic CS jobs, including some from Canada.

References

1. N. Van Houdnos, S. Moon, D. French, Brian Lindauer, P. Jansen, J. Carbonell, C. Hines, W. Casey. “Human-Computer Decision Systems for Cybersecurity”, Presentation. https://resources.sei.cmu.edu/asset_files/Presentation/2016_017_001_474277.pdf
2. Borg, M., Lennerstad, I., Ros, R., Bjarnasson, E. “On Using Active Learning and Self-Training When Mining Performance Discussions on Stack Overflow”, arXiv:1705.02395v1. Preprint of paper accepted for the Proc. of the 21st International Conference on Evaluation and Assessment in Software Engineering, 2017.

What are you trying to do

I would like to sail to India and bring back gold and spices for the Crown of Spain.

How is it done today

Currently no one has sailed west. Everyone takes the trip east, around the Cape of Good Hope. Most of these people think the world is flat and that heading west would cause us to fall into space.

What’s new in your approach

I will head west. I’m pretty sure the Earth is round, and we can reach India from the west in less time

Who cares?

A faster trading route to India, monopolized by our mapping skills, would generate 1 million Real a month for the royal treasury.

Risks

There is a lot unknown about the middle of the Atlantic, including rumors from the Vikings that some colder land is in between. My math may be off in calculating the circumference of the Earth. I am not a great sailor. We may encounter fierce alien tribes.

Cost and schedule

For 1000 Real we can outfit four boats with sailors, supplies, and weapons (note: of course Columbus would never get all he requested, either!). We plan on a quick 1 year voyage to India, and one more year back.

Checkpoints for success

We plan to see India after 2000 nautical miles of sailing. While measuring distance at sea is currently impossible, after 3 months we expect to sight land. If not, we will head back.

Purpose

Software is an engineering discipline, for most definitions of ‘engineering’. My definition, for what it’s worth, includes the notion that it involves working on real systems that do things, and to that end research in software engineering can be seen as a design science, where the chief task is to “design and investigate artifacts in context”.² This implies that for the most part researchers in this space need to concern themselves with pragmatics: how will this work at scale? How do people do this now? What data can we use that has practical relevance?

However, traditional conference submissions (the dominant form of scholarly dissemination in Computer Science) tend to follow the 10page, aim/motivation/observations/conclusions framework, often full of Greek letters and references to obscure papers. Whether this is a good way to advance the engineering discipline is debatable, but in any event, such a submission tends to ignore two things: one, how people dealing with problems in practice can use the work; two, the artefacts related to the scientific endeavor (the ‘treatment’ in Wieringa’s design science parlance). While improving, too many research papers still do not include tool downloads, fail to show practical impact, or fail to provide for data download to replicate the findings.

Our engineering track is out to improve the practical, engineering-relevant side of source code analysis and manipulation.

Submission types

This track has evolved from the tool track of previous SCAMs. As David mentions,

This is not to discourage tool paper submissions–they will now fall into the Engineering Track–but to broaden the scope of the tools track … for those of you that invest blood, sweat, and tears into tooling, infrastructure, or realistic field studies SCAM recognizes the value of this work, which is not always pure research, and we are designing this track to attract that type of work.

What artefacts qualify as “engineering track” material (from CFP)?

• tools: software (or hardware!) programs that facilitate SCAMmy activities.

• libraries: reusable API-enabled frameworks for the above.

• infrastructure: while libraries are purely software, infrastructure can include projects that provide/facilitate access to data and analysis.

• data: reusable datasets for other researchers to replicated and innovate with.

• real world studies enabled by these advances. Here the focus is on how the {tool,infrastructure, etc} enabled the study, and not so much the study itself. Novelty of the research question is less important than the engineering challenges faced in the study.

Some of the criteria the PC will look at includes:

• How well motivated are the use cases (and hence the existence) of the engineering work. Here we are asking whether this solves some realistic and ongoing challenge in practice. However, we are open to brilliant new ideas that scratch a previously unknown itch³.
• Relate the engineering project to earlier work. All engineering is a product of lessons learned, so including some narrative about how this particular submission has evolved is useful (e.g., what paths turned out to be dead ends).

Optionally (and encouraged):

• Any empirical results or user feedback is welcome.
• Contain the URL of a website where the tool/library/data etcetera can be downloaded, together with example data and installation guidelines, preferably but not necessarily open source
• Contain the URL to a video demonstrating the usage of the contribution.

Ideally one would submit and make public the artifacts and required steps to create it. However, realistically people may not be able to (given IP rules, NDAs, etc.).

Program Committee

Building on SCAM general chair David Shepherd’s excellent blog post on industry tracks, both Jurgen and I are committed to a program committee (PC) that has strong industry representation. That doesn’t mean only people who work in industry, but at least means people who have some sense of the engineering challenges of building real-world software. The purpose is to vet submissions against the standards industry holds: not necessarily will work right away at scale in mission critical systems, but that there is some promise of that.

Incidentally, if you are a former academic now practicing, or just a research-minded practitioner, I would love to hear from you for future PCs. We need more folks straddling the two cultures.

“Related Work”

We are not the only place thinking of how to expand and include more non-traditional research papers. At MSR Working Conference on Mining Software Repositories there is a data track, a tools track, and a mining challenge.

One of my favorite venues, the International Conference on Requirements Engineering, has long had what I have found to be the strongest industry focus of any software conference. In part I think this is because RE is implicitly concerned with what business needs, but it also reflects a purposeful ambition to increase relevance of the research results. For example, there is a “Ready-Set-Transfer!” panel in which academics present tools to practitioners to review practical readiness.

Practitioner conferences are (almost by definition) industry focused⁴ and both the Agile series of conferences and the XP conference include mirror-world ‘research’ tracks.