To Örebro University

Background: Classifying natural language requirements (NLRs) is challenging, especially with large volumes. Research shows that Large Language Models (LLMs) can assist by categorising NLRs into Functional Requirements (FR) and Non-Functional Requirements (NFR). However, concerns about their consistency have been raised.

Aim: In this work, we aim to investigate the consistency when Generative Pretrained Transformer (GPT) -like models, such as GPT-4.1 and LLAMA3.3-70B, classify NLRs into FRs and NFRs by using the Few-Shot Learning (FSL) approach. We explore how the number of examples used in the FSL approach and the temperature setting affect the consistency of NRL classification results.

Method: For this study, we use the PROMISE NFR dataset. Experimental design is employed to investigate the binary classification of FR versus NFR. To accomplish this goal, we use FSL with GPT-4.1 and LLAMA3.3-70B to classify the requirements.

Results: The results show that GPT-4.1 performs similarly with two and ten examples. However, the results somewhat differ for LLAMA3.3-70B, where FSL is slightly better using FSL with ten examples compared to two. Moreover, both LLM models seem to perform well in identifying FRs but have a more conservative classification behaviour when classifying NFRs. Finally, the temperature parameter does not seem to significantly impact the classification results.

Conclusions: When using FSL, using two versus ten examples slightly impacts the results. Moreover, we experimented with different temperatures (Temp = 0, Temp = 1, Temp = 2), and the temperature parameter does not seem to impact the classification results. Therefore, LLMs do not seem to become more creative even when the temperature increases.

Generative Artificial Intelligence (GenAI) is increasingly integrated into software products to enable new features and user capabilities, from early exploration to operational deployment. GenAI adoption as a component within a software system introduces quality risks because GenAI outputs are probabilistic, prompt-sensitive, and may drift after release. Organizations, therefore, need to decide what to evaluate, when to evaluate, and who owns quality evaluation activities across software design, development, and operations. ISO/IEC 25059 standard distinguishes between software product quality (e.g., usability) and quality-in-use (e.g., satisfaction) for AI-enabled software, yet it provides limited operational guidance for these evaluation activities. We therefore investigate how industrial software teams adopt and use GenAI models in the software systems they build and operate, and how they evaluate system qualities when deciding to adopt GenAI during development and after deployment. We do not benchmark the underlying GenAI model itself. In this study, we conducted 19 semi-structured interviews in two software development companies. We triangulated the interviews with archival data (15 internal documents and 184 internal wiki/web pages) to capture GenAI adoption steps, quality concerns, evaluation practices, and role responsibilities. Our findings describe a three-phase adoption process - Ideation, Development, and Operation - highlighting where quality evaluations occur, which criteria are used, and how evaluation responsibilities are distributed. Based on observed practices and using ISO/IEC 25059 as an organizing lens, we synthesize a process-oriented quality evaluation framework. This framework maps metrics to explicit gatekeeping, validation, and monitoring checkpoints, bridging abstract ISO quality characteristics with engineering workflows. We applied the framework in a GenAI-enabled software product (SE4AI) use case and reported how it supported structured evaluation activities. We also observed that quality evaluations span legal, security, development, QA, and operations, but ownership is fragmented across phases. We therefore propose a GenAI Quality Lead responsibility (often assignable to an existing senior role) to coordinate criteria, evidence, and traceability across quality evaluation activities. The results contribute to Software Engineering for AI (SE4AI) by clarifying how teams can measure qualities when building software that adopts and uses GenAI.

In the postpandemic era, attitudes toward remote work appeared to undergo a lasting transformation, with a high degree of location flexibility becoming increasingly common. Yet, in recent years, many organizations have introduced return-to-office (RTO) initiatives aimed at re-establishing traditional workplace dynamics and prioritizing in-person collaboration. These mandates have drawn significant attention and criticism for limiting software developers flexibility, diminishing well-being, and potentially impacting women disproportionally. This study seeks to understand software developers preferences and actual work behaviors in companies that promote in-office presence. Specifically, we investigate whether certain demographic groups, including women, are differentially affected by RTO initiatives. We also explore a range of factors that may influence individual preferences for remote or on-site work, beyond gender-based assumptions. We report findings from a survey conducted in two large Scandinavian companies engaged in the development of software-intensive systems and services. Data analysis includes descriptive statistics, contingency tables along with post hoc tests, chi-square test of association, and Cram & eacute;r's for effect sizes. Our findings reveal that gender differences among software developers in both industrial cases are minimal and statistically insignificant. Instead, other variables-such as the degree of collaborative work, commute time, and responsibility to support teammates-demonstrate a stronger association with both actual and preferred office attendance. Our results challenge common narratives around gendered responses to RTO mandates, suggesting that other contextual and task-related factors may play a more decisive role. While the impact of RTO initiatives should not be dismissed, our findings indicate that a deeper understanding of work dynamics-particularly around collaboration intensity and commuting burden-is essential to designing equitable and effective work policies. Finally, our findings imply that organizational recommendations for work location must go hand in hand with task design.

Gamification is a powerful instrument in the educational process if applied with care, considering the holistic learning experience. Literature has shown that using game-based mechanisms can motivate and engage students to participate actively in the learning process. This study investigates students’ motivation and engagement to participate in practical courses developed based on a holistic gamification approach. In this paper, we present the results of a case study in a Swedish university. In one of our practical courses, we designed and implemented gamification concepts (bonus points, real-life scenarios, and role-playing). The course Software Architecture was designed and developed from scratch with the explicit aim of integrating gamification into its structure and delivery. After each lecture, we planned a workshop (with voluntary participation) with gamification elements, applying the learning outcomes. To investigate students’ insights, we distributed a research survey, complementing the general course evaluation survey, at the end of the course, during which 60% of the enrolled students were present (enrolled students:100). In total, we collected 50 responses. The preliminary results show positive statistical significance on the students’ motivation to participate actively and engage in the course; moreover, it encourages positive collaboration among the students and helps students better understand the course content. However, students experience different gamification elements (e.g., bonus points and role-playing) differently, i.e., role-playing seems to make students struggle more to apply it to a real-case scenario. Moreover, there are indications that the implemented gamification approach added extra stress and pressure to the students. We also identify implications that awarding bonus points, which are collected during seminars and used on the final exam, at the group level instead of an individual level, cannot be a fair approach. Overall, the results indicated that the students are motivated and engaged to participate in courses developed with a holistic gamification approach. The study will be replicated with a different group of students and in a theoretical course to better understand how gamification designs can be applied in different types of courses.

Work in software development companies has become increasingly hybrid with employees altering days of working in the office with days of working remotely from home. Yet, little is know about the efficiency of such way of working because the current scale of remote working is unprecedented. In this paper, we present our findings from a company-wide survey at Storebrand - a large-scale Norwegian fintech company, focusing on perceived performance. Our analysis of 192 responses shows that most employees report being able to perform the planned tasks. Further, half of respondents perceive to have increased work hours. Through qualitative analysis of open-ended commentaries of respondents we learned that remote working has dual effects on the perceived work hours - some employees report working longer hours and others report having more work time due to efficient use of the time throughout the day. Finally, we recommend managers to discuss and address the concerning habits of employees caused by increased connectivity and inability to stop working, before these lead to burnout and disturbances in the work/life balance.

Context: Generative AI (GenAI) is increasingly adopted in software development for tasks such as document generation, data analysis, and code generation. However, evaluating the quality of GenAI applications becomes challenging, as traditional quality measurements may not be fully applicable.

Objective: In this study, we explore how practitioners evaluate the quality of GenAI applications and investigate quality evaluation techniques.

Method: We conducted a multi-case study in three industrial projects from software development companies. We examined four GenAI application domains: document generation, data analysis and insight generation, customer service, and code generation. Data were collected through three workshops and 23 semi-structured interviews with industrial practitioners.

Results: We identified fourteen GenAI use cases and 28 metrics currently used to evaluate the quality of GenAI applications' outputs. We synthesized the identified metrics' usage patterns and challenges based on the collected data.

Conclusions: This study presents practical insights into using metrics to measure GenAI-based system qualities in real industrial settings. Our findings indicate that practitioners use custom-built and context-specific metrics; combining these with academic metrics can strengthen GenAI system quality evaluation.

Classifying natural language requirements (NLRs) is challenging, especially with large volumes. Research shows that Large Language Models can assist by categorizing NLRs into functional requirements (FR) and non-functional requirements (NFRs). However, Generative Pretrained Transformer (GPT) models are not typically favored for this task due to concerns about consistency. This paper investigates the consistency when a GPT model classifies NLRs into FRs and NFRs using a zero-shot learning approach. Results show that ChatGPT-4o performs better for FRs, a temperature parameter set to 1 yields the highest consistency, while NFR classification improves with higher temperatures.

Classifying natural language requirements (NLRs) plays a crucial role in software engineering, helping us distinguish between functional and non-functional requirements. While large language models offer automation potential, we should address concerns about their consistency, meaning their ability to produce the same results over time. In this work, we share experiences from experimenting with how well GPT-4o and LLAMA3.3-70B classify NLRs using a zero-shot learning approach. Moreover, we explore how the temperature parameter influences classification performance and consistency for these models. Our results show that large language models like GPT-4o and LLAMA3.3- 70B can support automated NLRs classification. GPT-4o performs well in identifying functional requirements, with the highest consistency occurring at a temperature setting of one. Additionally, non-functional requirements classification improves at higher temperatures, indicating a trade-off between determinism and adaptability. LLAMA3.3-70B is more consistent than GPT-4o, and its classification accuracy varies less depending on temperature adjustments.

Background: Multi-label requirements classification is an inherently challenging task, especially when dealing with numerous classes at varying levels of abstraction. The task becomes even more difficult when a limited number of requirements is available to train a supervised classifier. Zero-shot learning does not require training data and can potentially address this problem.

Objective: This paper investigates the performance of zero-shot classifiers on a multi-label industrial dataset. The study focuses on classifying requirements according to a hierarchical taxonomy designed to support requirements tracing.

Method: We compare multiple variants of zero-shot classifiers using different embeddings, including 9 language models (LMs) with a reduced number of parameters (up to 3B), e.g., BERT, and 5 large LMs (LLMs) with a large number of parameters (up to 70B), e.g., Llama. Our ground truth includes 377 requirements and 1968 labels from 6 output spaces. For the evaluation, we adopt traditional metrics, i.e., precision, recall, F-1, and F-beta, as well as a novel label distance metric D-n. This aims to better capture the classification's hierarchical nature and to provide a more nuanced evaluation of how far the results are from the ground truth.

Results: 1) The top-performing model on 5 out of 6 output spaces is T5-xl, with maximum F-beta = 0:78 and D-n = 0:04, while BERT base outperformed the other models in one case, with maximum F-beta = 0:83 and D-n = 0:04. 2) LMs with smaller parameter size produce the best classification results compared to LLMs. Thus, addressing the problem in practice is feasible as limited computing power is needed. 3) The model architecture (autoencoding, autoregression, and sentence-to-sentence) significantly affects the classifier's performance.

Contribution: We conclude that using zero-shot learning for multi-label requirements classification offers promising results. We also present a novel metric that can be used to select the top-performing model for this problem.

Context: Generative Artificial Intelligence (GenAI) and the use of Large Language Models (LLMs) have revolutionized tasks that previously required significant human effort, which has attracted considerable interest from industry stakeholders. This growing interest has accelerated the integration of AI models into various industrial applications. However, the model integration introduces challenges to product quality, as conventional quality measuring methods may fail to assess GenAI systems. Consequently, evaluation techniques for GenAI systems need to be adapted and refined. Examining the current state and applicability of evaluation techniques for the GenAI system outputs is essential.

Objective: This study aims to explore the current metrics, methods, and processes for assessing the outputs of GenAI systems and the potential of risky outputs.

Method: We performed a snowballing literature review to identify metrics, evaluation methods, and evaluation processes from 43 selected papers.

Results: We identified 28 metrics and mapped these metrics to four quality characteristics defined by the ISO/IEC 25023 standard for software systems. Additionally, we discovered three types of evaluation methods to measure the quality of system outputs and a three-step process to assess faulty system outputs. Based on these insights, we suggested a five-step framework for measuring system quality while utilizing GenAI models.

Conclusion: Our findings present a mapping that visualizes candidate metrics to be selected for measuring quality characteristics of GenAI systems, accompanied by step-by-step processes to assist practitioners in conducting quality assessments.