To Örebro University

Gamification is a powerful instrument in the educational process if applied with care, considering the holistic learning experience. Literature has shown that using game-based mechanisms can motivate and engage students to participate actively in the learning process. This study investigates students’ motivation and engagement to participate in practical courses developed based on a holistic gamification approach. In this paper, we present the results of a case study in a Swedish university. In one of our practical courses, we designed and implemented gamification concepts (bonus points, real-life scenarios, and role-playing). The course Software Architecture was designed and developed from scratch with the explicit aim of integrating gamification into its structure and delivery. After each lecture, we planned a workshop (with voluntary participation) with gamification elements, applying the learning outcomes. To investigate students’ insights, we distributed a research survey, complementing the general course evaluation survey, at the end of the course, during which 60% of the enrolled students were present (enrolled students:100). In total, we collected 50 responses. The preliminary results show positive statistical significance on the students’ motivation to participate actively and engage in the course; moreover, it encourages positive collaboration among the students and helps students better understand the course content. However, students experience different gamification elements (e.g., bonus points and role-playing) differently, i.e., role-playing seems to make students struggle more to apply it to a real-case scenario. Moreover, there are indications that the implemented gamification approach added extra stress and pressure to the students. We also identify implications that awarding bonus points, which are collected during seminars and used on the final exam, at the group level instead of an individual level, cannot be a fair approach. Overall, the results indicated that the students are motivated and engaged to participate in courses developed with a holistic gamification approach. The study will be replicated with a different group of students and in a theoretical course to better understand how gamification designs can be applied in different types of courses.

Work in software development companies has become increasingly hybrid with employees altering days of working in the office with days of working remotely from home. Yet, little is know about the efficiency of such way of working because the current scale of remote working is unprecedented. In this paper, we present our findings from a company-wide survey at Storebrand - a large-scale Norwegian fintech company, focusing on perceived performance. Our analysis of 192 responses shows that most employees report being able to perform the planned tasks. Further, half of respondents perceive to have increased work hours. Through qualitative analysis of open-ended commentaries of respondents we learned that remote working has dual effects on the perceived work hours - some employees report working longer hours and others report having more work time due to efficient use of the time throughout the day. Finally, we recommend managers to discuss and address the concerning habits of employees caused by increased connectivity and inability to stop working, before these lead to burnout and disturbances in the work/life balance.

Context: Generative AI (GenAI) is increasingly adopted in software development for tasks such as document generation, data analysis, and code generation. However, evaluating the quality of GenAI applications becomes challenging, as traditional quality measurements may not be fully applicable.

Objective: In this study, we explore how practitioners evaluate the quality of GenAI applications and investigate quality evaluation techniques.

Method: We conducted a multi-case study in three industrial projects from software development companies. We examined four GenAI application domains: document generation, data analysis and insight generation, customer service, and code generation. Data were collected through three workshops and 23 semi-structured interviews with industrial practitioners.

Results: We identified fourteen GenAI use cases and 28 metrics currently used to evaluate the quality of GenAI applications' outputs. We synthesized the identified metrics' usage patterns and challenges based on the collected data.

Conclusions: This study presents practical insights into using metrics to measure GenAI-based system qualities in real industrial settings. Our findings indicate that practitioners use custom-built and context-specific metrics; combining these with academic metrics can strengthen GenAI system quality evaluation.

Classifying natural language requirements (NLRs) is challenging, especially with large volumes. Research shows that Large Language Models can assist by categorizing NLRs into functional requirements (FR) and non-functional requirements (NFRs). However, Generative Pretrained Transformer (GPT) models are not typically favored for this task due to concerns about consistency. This paper investigates the consistency when a GPT model classifies NLRs into FRs and NFRs using a zero-shot learning approach. Results show that ChatGPT-4o performs better for FRs, a temperature parameter set to 1 yields the highest consistency, while NFR classification improves with higher temperatures.

Classifying natural language requirements (NLRs) plays a crucial role in software engineering, helping us distinguish between functional and non-functional requirements. While large language models offer automation potential, we should address concerns about their consistency, meaning their ability to produce the same results over time. In this work, we share experiences from experimenting with how well GPT-4o and LLAMA3.3-70B classify NLRs using a zero-shot learning approach. Moreover, we explore how the temperature parameter influences classification performance and consistency for these models. Our results show that large language models like GPT-4o and LLAMA3.3- 70B can support automated NLRs classification. GPT-4o performs well in identifying functional requirements, with the highest consistency occurring at a temperature setting of one. Additionally, non-functional requirements classification improves at higher temperatures, indicating a trade-off between determinism and adaptability. LLAMA3.3-70B is more consistent than GPT-4o, and its classification accuracy varies less depending on temperature adjustments.

Background: Multi-label requirements classification is an inherently challenging task, especially when dealing with numerous classes at varying levels of abstraction. The task becomes even more difficult when a limited number of requirements is available to train a supervised classifier. Zero-shot learning does not require training data and can potentially address this problem.

Objective: This paper investigates the performance of zero-shot classifiers on a multi-label industrial dataset. The study focuses on classifying requirements according to a hierarchical taxonomy designed to support requirements tracing.

Method: We compare multiple variants of zero-shot classifiers using different embeddings, including 9 language models (LMs) with a reduced number of parameters (up to 3B), e.g., BERT, and 5 large LMs (LLMs) with a large number of parameters (up to 70B), e.g., Llama. Our ground truth includes 377 requirements and 1968 labels from 6 output spaces. For the evaluation, we adopt traditional metrics, i.e., precision, recall, F-1, and F-beta, as well as a novel label distance metric D-n. This aims to better capture the classification's hierarchical nature and to provide a more nuanced evaluation of how far the results are from the ground truth.

Results: 1) The top-performing model on 5 out of 6 output spaces is T5-xl, with maximum F-beta = 0:78 and D-n = 0:04, while BERT base outperformed the other models in one case, with maximum F-beta = 0:83 and D-n = 0:04. 2) LMs with smaller parameter size produce the best classification results compared to LLMs. Thus, addressing the problem in practice is feasible as limited computing power is needed. 3) The model architecture (autoencoding, autoregression, and sentence-to-sentence) significantly affects the classifier's performance.

Contribution: We conclude that using zero-shot learning for multi-label requirements classification offers promising results. We also present a novel metric that can be used to select the top-performing model for this problem.

Context: Generative Artificial Intelligence (GenAI) and the use of Large Language Models (LLMs) have revolutionized tasks that previously required significant human effort, which has attracted considerable interest from industry stakeholders. This growing interest has accelerated the integration of AI models into various industrial applications. However, the model integration introduces challenges to product quality, as conventional quality measuring methods may fail to assess GenAI systems. Consequently, evaluation techniques for GenAI systems need to be adapted and refined. Examining the current state and applicability of evaluation techniques for the GenAI system outputs is essential.

Objective: This study aims to explore the current metrics, methods, and processes for assessing the outputs of GenAI systems and the potential of risky outputs.

Method: We performed a snowballing literature review to identify metrics, evaluation methods, and evaluation processes from 43 selected papers.

Results: We identified 28 metrics and mapped these metrics to four quality characteristics defined by the ISO/IEC 25023 standard for software systems. Additionally, we discovered three types of evaluation methods to measure the quality of system outputs and a three-step process to assess faulty system outputs. Based on these insights, we suggested a five-step framework for measuring system quality while utilizing GenAI models.

Conclusion: Our findings present a mapping that visualizes candidate metrics to be selected for measuring quality characteristics of GenAI systems, accompanied by step-by-step processes to assist practitioners in conducting quality assessments.

The quality of the test suites and the constituent test cases significantly impacts confidence in software testing. While research has identified several quality attributes of test cases and test suites, there is a need for a better understanding of their relative importance in practice. We investigate practitioners' perceptions regarding the relative importance of quality attributes of test cases and test suites and the challenges that they face in ensuring the perceived important quality attributes. To capture the practitioners' perceptions, we conducted an industrial survey using a questionnaire based on the quality attributes identified in an extensive literature review. We used a sampling strategy that leverages LinkedIn to draw a large and heterogeneous sample of professionals with experience in software testing. We collected 354 responses from practitioners with a wide range of experience (from less than one year to 42 years of experience). We found that the majority of practitioners rated Fault Detection, Usability, Maintainability, Reliability, and Coverage to be the most important quality attributes. Resource Efficiency, Reusability, and Simplicity received the most divergent opinions, which, according to our analysis, depend on the software-testing contexts. Also, we identified common challenges that apply to the important attributes, namely inadequate definition, lack of useful metrics, lack of an established review process, and lack of external support. The findings point out where practitioners actually need further support with respect to achieving high-quality test cases and test suites under different software testing contexts. Hence, the findings can serve as a guideline for academic researchers when looking for research directions on the topic. Furthermore, the findings can be used to encourage companies to provide more support to practitioners to achieve high-quality test cases and test suites.

The pandemic experiences of forced work from home (WFH) in the tech industry turned out better than expected. As a result, modern workplaces that employ software engineers have become increasingly hybrid allowing employees to alternate days spent in the office with days spent working remotely. Yet, approaches to regulate the hybrid work arrangements vary. Some companies implemented strict policies with controlled office presence while others rely on recommendations and permit greater locational flexibility. In this paper, we evaluate how different degrees of locational flexibility influence individual work arrangements and satisfaction in two comparative cases: a company with high degree of flexibility (FinCo) and a company with mandatory office presence (TelCo). Through a survey of 547 practitioners, our findings reveal that flexible policies can achieve higher voluntary office attendance than mandatory requirements. To our surprise, we found that the number of employees visiting the office at least 2-3 days per week in the company with greater flexibility was higher (68%) compared to the company with mandatory attendance (58%). The study also identifies key factors influencing work location choices, including commute time and role. The type of tasks and dependencies with colleagues also matter - employees with more WFH days tended to have more individual tasks, while those with more onsite work days engaged in more collaborative tasks and had colleagues who depended on them. Our results suggest that trust-based approaches and creating attractive office environments may be more effective than strict attendancepolicies in maintaining desired office presence while supporting employee satisfaction. These findings contribute practical insights for organizations seeking to establish effective post-pandemic wor kpolicies for software engineers.

Visualizing CI's role in software quality attribute evaluation.

QUALITY ATTRIBUTES OF software systems, also known as system qualities, such as performance, security, and scalability, continue to grow in importance in industrial practice. The evaluation of quality attributes is critical to software development since optimizing a software system's core attributes can provide marketing advantage and set a product apart from its competitors. Many existing studies of unsuccessful development projects report that lack of quality attribute evaluation is often a contributing factor of project failure. Therefore, continuous quality attribute evaluation, throughout the development process, is needed to ensure customers' expectations and demands are met.