"

Scoping review design and methods

Reflections on AI-assisted methods: Opportunities and limitations

 i.      AI-assisted thematic analysis

In addition to conventional reading and coding, we made exploratory use of large language models (LLMs) to support sense-making across our relatively large and heterogeneous corpus, as described above. Our intention was not to automate thematic analysis, but to use AI tools as heuristic devices that could surface potential patterns and contrasts for subsequent critical examination. As noted, we experimented with two LLM-based tools (ChatGPT 4.5 and SciSpace) to assist in generating and clustering candidate codes. In all cases, final thematic decisions were based on human interpretation of the source texts, with LLM outputs functioning as supplementary, not determinative, inputs.

AI assistance proved useful in two main ways. First, it helped us to scan for recurring topical combinations across the corpus, reducing the risk that isolated but thematically similar articles would be overlooked. Second, AI-suggested coding occasionally functioned as productive “provocations,” prompting us to revisit and refine our interpretive decisions.

LLMs are known, however, to produce hallucinated or over-generalized claims. To mitigate this, we did not accept any AI-generated description, code, or thematic label at face value. All candidate codes and clusters were cross-checked against the text of the respective articles, and no article was coded solely on the basis of model output.

Because commercial LLMs are trained on uneven and often opaque datasets, their suggestions may over-privilege Anglo-American terminology and framings. In this project the model’s scope was constrained to our pre-selected corpus, and it therefore did not determine which works were included or how “important” they were. Nonetheless, we remained alert to the possibility that AI-suggested code clusters might foreground familiar English-language labels or frameworks at the expense of regionally specific conceptualizations. Where this appeared to occur, we prioritized local terminology in our final thematic labels.

A further limitation is the tendency of LLMs to compress complex arguments into tidy topical categories. Given our interest in contested, processual understandings of culture, we treated AI-generated summaries and labels as heuristic starting points only. Instances where model outputs seemed to smooth over tensions or contradictions in the source texts were flagged for closer human reading, and we deliberately retained ambiguity in cases where the underlying literature was itself unsettled.

Because LLM outputs are non-deterministic and dependent on model version, the precise wording of AI-suggested codes cannot be reproduced identically. To enhance transparency, we archived the main prompts and a sample of model outputs as analytic memos. Our results, however, do not depend on any single AI generation: themes were derived from repeated human engagement with the corpus, and could be reconstructed independently of the specific model suggestions.

Finally, we restricted AI inputs to published bibliographic data and article text. No confidential or personally identifiable data were processed by the models, and our use of LLMs is consistent with emerging institutional guidelines on AI-assisted research workflows.

Our use of AI assistance is thus both pragmatic and reflexive. Working with LLMs during analysis made tangible several dynamics we later discuss in relation to the Fourth Age of model-mediated communication, including the salience of model defaults, tendencies toward stylistic normalization, and the risk that uneven training data may re-centre dominant discourses. By explicitly foregrounding these opportunities and limitations in our methods, we aim to model a critically engaged, accountable approach to AI-assisted scholarship.

ii.      AI-assisted writing

The use of generative AI in writing raises related, but distinct, concerns from its use in analysis. First, there is a risk that model-suggested formulations may normalize style and epistemic stance, flattening voice, discipline-specific nuance, or critical edge. We therefore treated AI-assisted editing as a way to surface potential alternatives rather than as a stylistic template, and we deliberately re-introduced nuance, detail, authorial voice, or disciplinary terminology as needed. Second, while LLMs can sometimes suggest plausible but inaccurate claims or citations, we restricted prompts to our own draft text and did not ask the model to “fill in” literature or factual content. All empirical claims, references, and interpretations of sources derive from our review of the underlying works and were checked against the original texts. Responsibility for the arguments and wording in the final manuscript rests fully with the human authors.

 

definition

License

Icon for the Creative Commons Attribution-NonCommercial 4.0 International License

Culture and Communication in Digital Worlds Copyright © 2025 by Leah P. Macfadyen is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, except where otherwise noted.

Share This Book