Evaluating the quality of GenAI applications in software engineering: a multi-case study
2025 (English)In: Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616, Vol. 31, no 2, article id 29Article in journal (Refereed) Published
Abstract [en]
Context: Generative AI (GenAI) is increasingly adopted in software development for tasks such as document generation, data analysis, and code generation. However, evaluating the quality of GenAI applications becomes challenging, as traditional quality measurements may not be fully applicable.
Objective: In this study, we explore how practitioners evaluate the quality of GenAI applications and investigate quality evaluation techniques.
Method: We conducted a multi-case study in three industrial projects from software development companies. We examined four GenAI application domains: document generation, data analysis and insight generation, customer service, and code generation. Data were collected through three workshops and 23 semi-structured interviews with industrial practitioners.
Results: We identified fourteen GenAI use cases and 28 metrics currently used to evaluate the quality of GenAI applications' outputs. We synthesized the identified metrics' usage patterns and challenges based on the collected data.
Conclusions: This study presents practical insights into using metrics to measure GenAI-based system qualities in real industrial settings. Our findings indicate that practitioners use custom-built and context-specific metrics; combining these with academic metrics can strengthen GenAI system quality evaluation.
Place, publisher, year, edition, pages
Springer, 2025. Vol. 31, no 2, article id 29
Keywords [en]
Generative artificial intelligence, GenAI, Large language model, LLM, Metric, Quality evaluation
National Category
Artificial Intelligence
Identifiers
URN: urn:nbn:se:oru:diva-126190DOI: 10.1007/s10664-025-10759-2ISI: 001632325800004Scopus ID: 2-s2.0-105024070431OAI: oai:DiVA.org:oru-126190DiVA, id: diva2:2028988
Funder
Blekinge Institute of Technology2026-01-162026-01-162026-01-23Bibliographically approved