Is GenAI a Silver Bullet for Qualitative Data Analysis?
TL;DR
I wrote an opinion piece with my friend Christoph Treude on qualitative data analysis (QDA) in the GenAI era. We conclude that it has great capability in speeding up the process (e.g., transcription, finding codes) but is fundamentally and philosophically at odds with Big-Q reflexive data analysis.
Below I elaborate on my perspective, which Christoph may not agree with!
- Paper: https://doi.org/10.1145/3837087
- Replication: Zenodo
Reflexivity in the Age of GenAI
Our piece combines a short survey of three venues with background on QDA and research quality. We found that very few papers in 2025 are using GenAI to replace human coders. The main use was to employ the AI as a judge in a deductive, post-positivist epistemological stance.1 These include tasks like “label this code commit” or “use this coding guide to interpret this pull request review”. The main output is counting occurrences of labels or codes.
I fully expect this usage will evolve in 2026 and onward. The seductive appeal of a tool that can take a labor-intensive task (like, hundreds of hours) and reduce that to minutes, will be too tempting in an era of publish (10s of papers a year) or perish.
The big distinction I draw is between what has been called small-q and Big-Q qualitative research. The Big-Q approach (from Kidder and Fine, see paper for more) focuses on reflexivity, induction, and interpretivist epistemology. It centers the human interpreter and their judgment of the data, the research goals, and the knowledge gained therein. This would include interview studies of students using GenAI tools in coursework, for example.
For this type of work, AI use to derive the themes is antithetical. The doyennes of reflexive thematic analysis, Braun and Clarke, contributed to a fairly polemical editorial on this topic.
The Role of Abduction
Ultimately Big-Q reflexive QDA is a creative process, and I have yet to be convinced statistical language models can be creative. They are excellent at deduction - following patterns to their conclusion, as with AlphaProof; or at induction, distilling summaries from lots of observations. But, as argued in the essay LLMs Can’t Jump, they lack the sensory experience necessary to come to an abductive inference to new theories. Sensory experience in this context might involve having sat and interviewed respondents, seen their body language, been in their workplace, etc.
The Way Forward, or What Should I Do?
First and foremost, it would be useful if papers explicitly committed to a philosophical perspective. Too often the field defaults to post-positivist approaches, and never questions how the authors see knowledge, or what exists in the world.
We still need more understanding of the following:
- Benchmarking GenAI in coder-like roles. How effective is it? What rater agreements do we see, and where is the variance concentrated?
- Extending GenAI evaluation to interpretive methods. Most studies focus on deductive tasks (summarize this code snippet).
- Designing collaborative human-AI workflows. GenAI is being integrated into many QDA tools (e.g., NVivo). What is the most effective workflow? Where do humans belong?
- Developing reporting and evaluation norms for SE research practice. While the GenAI guidelines we worked on touch on this, practice is changing fast, and developing a shared understanding of how to use and report and review is important.
- Reconciling GenAI and constructivist and interpretivist research paradigms. GenAI has a place in Big-Q research—I don’t think anyone will manually transcribe interviews anymore—but where that line should be drawn is also unclear.
Post-script
Just today I found the following essay on Mastodon, “Do It 14,000 Times Slower With This One Simple Trick. I think it sums up my thoughts on GenAI quite succinctly. For me, at least, part of the fun of research2 in qualitative spaces is the deep engagement with human reactions. Automating all of that to increase my paper output is answering the wrong question.