Toward Innovativeness in Narrativity Measurement: A Comparative Analysis of Human, AI, and Hybrid Approaches

This study evaluates the reliability and validity of AI chatbots based on Large Language Models (LLMs) in narrativity assessment by comparing human-only, AI-only, and hybrid coding models. Using an 8-item narrativity scale, we analysed 32 stimuli texts across three AI chatbots (ChatGPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro) and two human coders, generating a total of 160 coding points. Following a systematic refinement and calibration process, AI chatbots demonstrated substantial reliability gains, with the hybrid model (30% human, 70% AI) providing the most efficient and accurate results. A series of tests addressing convergent, predictive, and concurrent validity established preliminary evidence supporting the narrativity coding scale's construct reliability and applicability for narrative research. While human and AI coders generally aligned in evaluation trends, notable differences emerged: AI chatbots exhibited superior consistency in identifying structured narrative elements, such as temporal sequencing and motivation-action patterns, whereas human coders captured narrative holism and cultural nuances, underscoring each approach’s unique strengths. Narratives consistently scored higher on narrativity than non-narratives, with increased narrativity correlating with reduced resistance, supporting both the persuasive effects of narratives and the scale’s internal consistency. These findings highlight methodological considerations for AI integration in narrative research and demonstrate AI’s potential for systematic, nuanced analysis in social sciences.