The Use of Large Language Models in Qualitative Text Analysis: A Comparative Study of AI and Traditional Methods in Social Sciences on the Example of Women's Motivations for NGO Participation
Aneta Uss-Lik1 ✉️, Berenika Dyczek2 ✉️ 1University of Wroclaw, Poland 2University of Wroclaw, Poland
Cite as: Uss-Lik, A. and Dyczek, B. (2025, May). The Use of Large Language Models in Qualitative Text Analysis: A Comparative Study of AI and Traditional Methods in Social Sciences on the Example of Women's Motivations for NGO Participation. In SCS 2025, 2nd International Conference on Social Contexts of Science (p. 42). Wrocław University of Science and Technology, Poland.
Abstract
This study focuses on the application of large language models (LLMs) such as GPT-4 (OpenAI), Claude (Anthropic), LLaMA-2 (Meta), Mistral, Gemini (Google), and Polish AI systems, including PLLuM and Bielik, for analyzing in-depth interviews (IDI) conducted with women regarding their motivations for participating in non-governmental organizations (NGOs). The primary objective is to compare the effectiveness of LLMs in linguistic data analysis with traditional qualitative research methods, evaluate their potential and limitations, and outline future directions for developing AI-supported tools for text analysis in the social sciences. The study adopts a computational social science approach, combining AI-driven text analysis with traditional qualitative research methods. The methodology involves processing a dataset of interviews to identify key motivations, barriers, and reflections shared by participants. LLMs are assessed in terms of their ability to categorize responses, analyze sentiment, detect linguistic patterns, interpret content within situational and cultural contexts, and apply data clustering and categorization methods. The results obtained through LLM-based analysis are compared with findings from classical qualitative research to evaluate the reliability and effectiveness of both approaches. The findings highlight both the strengths and limitations of using LLMs for text analysis. While these models enable rapid and scalable processing of large datasets, their ability to capture subtle cultural contexts, emotions, and linguistic nuances remains constrained compared to traditional research methods. The study also addresses key ethical considerations, including privacy concerns, algorithmic biases, and the risk of misinterpreting participant voices. The final evaluation of LLMs in the context of text analysis in the social sciences provides insights into their future development and applications. The study emphasizes the necessity of hybrid approaches that integrate advanced NLP techniques with human-driven qualitative interpretation. Additionally, it underscores the need for further research into improving LLMs’ ability to analyze social data and the development of AI-assisted tools that support researchers in exploring qualitative texts in a transparent and reliable manner.
Keywords
Large language models, Text analysis, AI in social sciences, Comparison of analytical methods, Contextual interpretation, Computational social sciences, AI ethics, Future of research tools
Current status of the research is: Work-in-progress