Technical Debt in Research Software

Neil Ernst

2024-10-11

There Isn’t Any

thank you and goodbye

About Me

  • Associate professor in Computer Science, University of Victoria (about 3 hrs by passenger ferry from Seattle)
  • Four years consulting/research at Software Engineering Institute, CMU/Pittsburgh (an FFRDC)
  • Research: software design in the LLM era, software research for/on research software, analytical sovereignty
  • Experiences: working with FDA on modernization, with the Square Kilometer Array on long-term design, with Canadian Space Agency on remote sensing tools.

Outline

  1. What is TD?
  2. Knowledge Domains
  3. Self-Admitted TD in Research Code
  4. Perceptions of TD
  5. Takeaways

Definitions / Poll

Cunningham

Ward Cunningham’s definition1: it’s good essential to go into a little bit of debt if it supports learning and is repaid

TD as a motivator/support for an agile and iterative approach to software development (1992!)

McConnell (and mine)

Technical debt occurs when a design or construction approach is taken that’s expedient in the short term, but that creates a technical context that increases complexity and cost in the long term.

Steve McConnell (Code Complete)

External/Internal

The source of the TD matters.

  • Internal TD: we did something to (learn more/get the paper published/funder said so) and now we have to fix it
  • External TD: someone did something and now my project has to be updated in non-interesting ways (i.e., exclude “new scientific findings”).
  • Now your fluid dynamics experts have suddenly also become security programmers because an authentication package is critical!1

But …

  • If we make a responsible decision in 1995 (e.g., for gridding by lat/lon) is it really TD if that becomes unsuitable in 2024?
  • In some ways it doesn’t matter! Maybe as a process check - are we tracking these impactful choices and consequences.
  • Worst thing to do is make design choices and not know why they were made
  • Debt is leverage: every code base should have some (~10-20%)

Knowledge Domains

Research software development requires cross-domain knowledge.

Domain (real-world) and Science (theory) knowledge

  • What you get from undergrad / postgrad work.
  • What papers get published on (mostly).

Software Knowledge

  • How to bring the theory and domain understanding into Fortran/Julia/Numpy
  • Bridge the finite, quantized, bit-oriented software world and the continuous, stochastic natural world

Execution Knowledge

  • Make the software work in a hardware and team environment
  • Deployment: cluster and hardware specifics. Makefiles, config files, XML.
  • Development: Git, GitHub, pull requests, code review.
  • Often not well taught in school, not well supported by tools
  • My feeling: the most under-appreciated domain

Operational Knowledge

  • Take the outcomes of the first four, and do something
    • data analysis, policy recommendations, new science
  • Crossing knowledge domains is a good way to inadvertently incur technical debt!

Knowledge Domains

Research software development requires cross-domain knowledge.

Self-Admitted Technical Debt in Research Software

Definition

the intentional acknowledgment of Technical Debt by developers through comments, commit messages, or documentation within the code.

# TODO: fix this poor approximation

Method

  • Choose 9 representative projects as samples
  • Identify 28,680 SATD comments using keyword searches and manual classification, ensuring accurate categorization.
  • Manually categorized comments using card sorting techniques

Projects

Project Scientific Domain Contribs. Project Type Code Size (KLOC) Age (Yrs)
Astropy Astronomy 453 Python Library 1,309 13
Athena High Energy Physics 100+ Software 5,208 19
Biopython Molecular Biology 331 Python Library 620 25
CESM Climate Model 134 Software 2,800 41
GROMACS Molecular Dynamics 85 Software 2,102 27
Moose Physics 221 Framework 848 16
Elmer Mathematics 45 Software 954 10
Firedrake Mathematics 96 Software 63 11
Root High Energy Physics 387 Framework 5,080 24

Scientific SATD

Accumulation of suboptimal scientific practices, assumptions, and inaccuracies within scientific software that potentially compromise the validity, accuracy, and reliability of scientific results.

Indicators

  • Translation Challenges
  • Assumptions
  • New Scientific Findings
  • Missing Edge Cases
  • Computational Accuracy

Translation

We are going to share n_eff between the neutrinos equally. In detail, this is not correct, but it is a standard assumption because properly calculating it is (a) complicated (b) depends on the details of the massive neutrinos (e.g., their weak interactions, which could be unusual if one is considering sterile neutrinos).”

Astropy

Assumptions

We assume here that new ice arrives at the surface with the same temperature as the surface. TODO: Make sure this assumption is consistent with energy conservation..

CESM

New Findings

Update the the instability calculation function and modify the neutral drag cofficient. We should follow more elegant approach like Louis et al (1982) or Bretherton and Park (2009) but for now we use very crude approach : just 1 for ri < 0, 0 for ri > 1, and linear ramping.

CESM

Missing Edge Cases

TODO(wjs, 2015-10-18) With the ‘qflx_snow_grnd_col(c) > 0.0_r8’ check in the following conditional, Leo van Kampenhout has found that, under some rare conditions, the snow pack does not get initialized when it should. However, if this check is removed, then under some different rare conditions, snow depth can grow infinitely. We clearly need a more robust check here, and/or some fixed logic elsewhere. But for now we’re going with the lesser of the two bugs - i.e., allowing the snow pack to not be initialized when it should under some conditions.

CESM

Accuracy

Results: f’’(0) never really seems to converge to more than about 4 digits? Not sure how much this really affects the solution we get for f itself

MOOSE

Patterns