To Örebro University

Finding a balance between testing goals and testing resources can be considered as a most challenging issue, therefore test optimization plays a vital role in the area of software testing. Several parameters such as the objectives of the tests, test cases similarities and dependencies between test cases need to be considered, before attempting any optimization approach. However, analyzing corresponding testing artifacts (e.g. requirement specification, test cases) for capturing the mentioned parameters is a complicated task especially in a manual testing procedure, where the test cases are documented as a natural text written by a human. Thus, utilizing artificial intelligence techniques in the process of analyzing complex and sometimes ambiguous test data, is considered to be working in different industries. Test scheduling is one of the most popular and practical ways to optimize the testing process. Having a group of test cases which are required the same system setup, installation or testing the same functionality can lead to a more efficient testing process. In this paper, we propose, apply and evaluate a natural language processing-based approach that derives test cases' similarities directly from their test specification. The proposed approach utilizes the Levenshtein distance and converts each test case into a string. Test cases are then grouped into several clusters based on their similarities. Finally, a set of cluster-based parallel test scheduling strategies are proposed for execution. The feasibility of the proposed approach is studied by an empirical evaluation that has been performed on a Telecom use-case at Ericsson in Sweden and indicates promising results.

Software testing is still heavily dependent on human judgment since a large portion of testing artifacts, such as requirements and test cases are written in a natural text by experts. Identifying and classifying relevant test cases in large test suites is a challenging and also time-consuming task. Moreover, to optimize the testing process test cases should be distinguished based on their properties, such as their dependencies and similarities. Knowing the mentioned properties at an early stage of the testing process can be utilized for several test optimization purposes, such as test case selection, prioritization, scheduling,and also parallel test execution. In this paper, we apply, evaluate, and compare the performance of two deep learning algorithmsto detect the similarities between manual integration test cases. The feasibility of the mentioned algorithms is later examined in a Telecom domain by analyzing the test specifications of five different products in the product development unit at Ericsson AB in Sweden. The empirical evaluation indicates that utilizing deep learning algorithms for finding the similarities between manual integration test cases can lead to outstanding results.

Finding a balance between meeting the testing goals and testing resources is always a challenging task. Therefore, employing Machine Learning (ML) techniques for test optimization purposes has received a great deal of attention. However, utilizing ML techniques requires frequently large volumes of data to obtain reliable results. Since the data gathering is hard and also expensive, reducing unnecessary failure or retest in a testing process might end up minimizing the testing resources. Final test yield is a proper performance metric to measure the potential risks influencing certain failure rates. Typically, production determines the yield’s minimum threshold based on an empirical value given by the subject matter experts. However, those thresholds cannot monitor the yield’s fluctuations beyond the acceptable thresholds, which might cause potential failures in consecutive tests. Furthermore, defining the empirical thresholds as either too tight or too loose in production is one of the main causes of yield dropping in the testing process. In this paper, we propose an ML-based solution that detects the divergent yield points based on the prediction and raises a flag depending on the yield class to the testers when a divergent point is above a data-driven threshold. This flexibility enables engineers to have a quantifiable tool to measure to what extend the different changes in the production process are affecting the product performance and execute actions before they occur. The feasibility of the proposed solution is studied by an empirical evaluation, which has been performed on a Telecom use-case at Ericsson in Sweden and tested in two of the latest radio technologies, 4G and 5G.

Finding a balance between meeting test coverage and minimizing the testing resources is always a challenging task both in software (SW) and hardware (HW) testing. Therefore, employing machine learning (ML) techniques for test optimization purposes has received a great deal of attention. However, utilizing machine learning techniques frequently requires large volumes of valuable data to be trained. Although, the data gathering is hard and also expensive, manual data analysis takes most of the time in order to locate the source of failure once they have been produced in the so-called fault localization. Moreover, by applying ML techniques to historical production test data, relevant and irrelevant features can be found using strength association, such as correlation- and mutual information-based methods. In this paper, we use production data records of 100 units of a 5G radio product containing more than 7000 test results. The obtained results show that insightful information can be found after clustering the test results by their strength association, most linear and monotonic, which would otherwise be challenging to identify by traditional manual data analysis methods.

The traditional approach of categorizing test results as “Pass” or “Fail” based on fixed thresholds can be labor-intensive and lead to dropping test data. This paper presents a framework to enhance the semi-automated software testing process by detecting deviations in executed data and alerting when anomalous inputs fall outside data-driven thresholds. In detail, the proposed solution utilizes classification with convolutional neural networks and prediction modeling using linear regression, Ridge regression, Lasso regression, and XGBoost. The study also explores transfer learning in a highly correlated use case. Empirical evaluation at a leading Telecom company validates the effectiveness of the approach, showcasing its potential to improve testing efficiency and accuracy. Despite its significance, limitations include the need for further research in different domains and industries to generalize the findings, as well as the potential biases introduced by the selected machine learning models. Overall, this study contributes to the field of semi-automated software testing and highlights the benefits of leveraging data-driven thresholds and machine learning techniques for enhanced software quality assurance processes.