Utilising CI environment for efficient and effective testing of NFRs
2020 (English)In: Information and Software Technology, ISSN 0950-5849, E-ISSN 1873-6025, Vol. 117, article id 106199Article in journal (Refereed) Published
Abstract [en]
Context
Continuous integration (CI) is a practice that aims to continuously verify quality aspects of a software intensive system both for functional and non-functional requirements (NFRs). Functional requirements are the inputs of development and can be tested in isolation, utilising either manual or automated tests. In contrast, some NFRs are difficult to test without functionality, for NFRs are often aspects of functionality and express quality aspects. Lacking this testability attribute makes NFR testing complicated and, therefore, underrepresented in industrial practice. However, the emergence of CI has radically affected software development and created new avenues for software quality evaluation and quality information acquisition. Research has, consequently, been devoted to the utilisation of this additional information for more efficient and effective NFR verification.
Objective
We aim to identify the state-of-the-art of utilising the CI environment for NFR testing, hereinafter referred to as CI-NFR testing.
Method
Through rigorous selection, from an initial set of 747 papers, we identified 47 papers that describe how NFRs are tested in a CI environment. Evidence-based analysis, through coding, is performed on the identified papers in this SLR.
Results
Firstly, ten CI approaches are described by the papers selected, each describing different tools and nine different NFRs where reported to be tested. Secondly, although possible, CI-NFR testing is associated with eight challenges that adversely affect its adoption. Thirdly, the identified CI-NFR testing processes are tool-driven, but there is a lack of NFR testing tools that can be used in the CI environment. Finally, we proposed a CI framework for NFRs testing.
Conclusion
A synthesised CI framework is proposed for testing various NFRs, and associated CI tools are also mapped. This contribution is valuable as results of the study also show that CI-NFR testing can help improve the quality of NFR testing in practices.
Place, publisher, year, edition, pages
Elsevier, 2020. Vol. 117, article id 106199
Keywords [en]
Agile, Continuous integration, CI, DevOps, Non-functional requirement, NFR, Scaled agile framework, SAFe
National Category
Information Systems Software Engineering
Identifiers
URN: urn:nbn:se:oru:diva-79434DOI: 10.1016/j.infsof.2019.106199ISI: 000496874400005Scopus ID: 2-s2.0-85073572821OAI: oai:DiVA.org:oru-79434DiVA, id: diva2:1388993
Funder
Knowledge Foundation
Note
We would like to acknowledge that this work was supported by the KKS foundation through the S.E.R.T. Research Profile project at Blekinge Institute of Technology.
2020-01-282020-01-282024-03-04Bibliographically approved