To Örebro University

oru.seÖrebro University Publications
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Evaluating the quality of GenAI applications in software engineering: a multi-case study
Blekinge Institute of Technology, Karlskrona, Sweden.
Blekinge Institute of Technology, Karlskrona, Sweden.
Örebro University, Örebro University School of Business.ORCID iD: 0000-0002-0311-1502
Blekinge Institute of Technology, Karlskrona, Sweden; Fortiss, Munich, Germany.
2025 (English)In: Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616, Vol. 31, no 2, article id 29Article in journal (Refereed) Published
Abstract [en]

Context: Generative AI (GenAI) is increasingly adopted in software development for tasks such as document generation, data analysis, and code generation. However, evaluating the quality of GenAI applications becomes challenging, as traditional quality measurements may not be fully applicable.

Objective: In this study, we explore how practitioners evaluate the quality of GenAI applications and investigate quality evaluation techniques.

Method: We conducted a multi-case study in three industrial projects from software development companies. We examined four GenAI application domains: document generation, data analysis and insight generation, customer service, and code generation. Data were collected through three workshops and 23 semi-structured interviews with industrial practitioners.

Results: We identified fourteen GenAI use cases and 28 metrics currently used to evaluate the quality of GenAI applications' outputs. We synthesized the identified metrics' usage patterns and challenges based on the collected data.

Conclusions: This study presents practical insights into using metrics to measure GenAI-based system qualities in real industrial settings. Our findings indicate that practitioners use custom-built and context-specific metrics; combining these with academic metrics can strengthen GenAI system quality evaluation.

Place, publisher, year, edition, pages
Springer, 2025. Vol. 31, no 2, article id 29
Keywords [en]
Generative artificial intelligence, GenAI, Large language model, LLM, Metric, Quality evaluation
National Category
Artificial Intelligence
Identifiers
URN: urn:nbn:se:oru:diva-126190DOI: 10.1007/s10664-025-10759-2ISI: 001632325800004Scopus ID: 2-s2.0-105024070431OAI: oai:DiVA.org:oru-126190DiVA, id: diva2:2028988
Funder
Blekinge Institute of TechnologyAvailable from: 2026-01-16 Created: 2026-01-16 Last updated: 2026-01-23Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Chatzipetrou, Panagiota

Search in DiVA

By author/editor
Chatzipetrou, Panagiota
By organisation
Örebro University School of Business
In the same journal
Empirical Software Engineering
Artificial Intelligence

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 45 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf