Research on Software Dynamic Random Testing: Theory and Its Application in Cloud Testing

動態隨機測試策略研究:理論及其在雲測試中的應用

Student thesis: Doctoral Thesis

View graph of relations

Author(s)

Detail(s)

Awarding Institution
Supervisors/Advisors
Award date20 Jan 2021

Abstract

With the development of computer science and increase in software sizes, software becomes more complex, and the losses caused by software faults becomes more serious. Excessive or serious faults may lead to a decline in software quality and rise the potentiality of software failure, especially for large scale software systems. Software testing is an important technology in ensuring software quality and improving software reliability, and it is a critical step in the whole software life cycle. One of the essences of software testing is to detect as much as possible faults with limited resources and ensure that the software meets the requirements. How to conduct testing in an effective way to ensure the software quality depends on the guidance of the testing theories. Besides, the development of cloud computing can provide a more effective software testing environment, which may further reduce the testing costs.

Dynamic Random Testing strategy (DRT) is a new testing strategy that follows the idea of software cybernetics and treats software testing as an issue of feedback control. More specifically, DRT uses the testing results as feedback information to adjust the testing profile dynamically in a concise way during the testing process, thus the selection probabilities of the subdomains with higher fault detection rates can be increased. This thesis reviewed the development and research status of DRT, and started with the improvement on the effectiveness and efficiency of DRT. Aiming at producing more efficient testing strategies and providing appropriate guidance for the application of DRT in practice, the influence factors on the performance of DRT, the improvement methods and the application in the cloud environment are studied. The research contents and contributions of this thesis are summarized as follows.

1. Investigation on fault detection effectiveness and efficiency of DRT: Some supplements to DRT are made, the performance of DRT with supplements, Random Testing (RT), Random Partition Testing strategy (RPT) and Adaptive Testing strategy (AT) are compared comprehensively. The experiment results show that DRT strategy can achieve remarkable better performance than RT and RPT without additional computational costs; and DRT can provide comparable effectiveness with AT while its computational cost is almost negligible compared with that of AT.

2. Analysis of influencing factors of DRT: The performance of DRT is mainly impacted by three factors, i.e., adjusting parameters, test case classification, and initial profiles. A series of experiments are conducted to investigate what influence the three factors may have on the performance of DRT. The experiment results show that the influence of the adjusting parameters is quite remarkable, and there exist an optimal area and a suitable ratio of the adjusting parameters for each subject software. Test case classification also has a noticeably influence on DRT, more specifically, the functional classification method and the combination classification method are the better methods to partition the test cases. And the initial profile may have a certain effect on the effectiveness of DRT, however, the effect should not be exaggerated. DRT is robust to different initial profiles by dynamically adjusting the testing profile.

3. Distance-based DRT (D-DRT): Since the performance of DRT is greatly affected by test case classification and the testing profile adjustment process, and the test case classification method used in the original DRT might not be desirable in some cases, a distance-based DRT strategy is proposed in this thesis. In D-DRT, the test cases are vectorized based on the input parameters, and partitioned according to a specific clustering method. And the distance information obtained from the test suite which can better reflect the discrepancies among the test cases is used to guide the adjustment of the testing profile, along with the feedback of the testing results. The experiment results demonstrate that D-DRT uses significantly fewer test cases than RT, RPT and the original DRT to detect a given number of faults without additional computational costs. Although the effectiveness of D-DRT is slightly inferior to AT, the computational cost of D-DRT is much lower than that of AT. It can be said that D-DRT strategies are better testing strategies in general. Besides, how do the test case clustering method and distance-based testing profile adjustment method improve the performance of DRT is investigated, and the results show that both methods can promote the performance of DRT to a certain extent, while the distance-based adjustment of the testing profile is the primary cause of improving the fault detection capability of DRT.

4. Cloud-based DRT: A cloud-based DRT strategy is proposed by combining cloud computing technology and DRT strategy. In the conventional DRT strategy, test cases are selected and executed in turn. In cloud-based DRT, the testing profile adjustment strategy in DRT is incorporated with the resource allocation strategy in cloud computing, and the test cases are selected and allocated dynamically in the testing process. While the testing profile is adjusted according to the feedbacks from the testing results, the time and resource costs are focused and the allocation process is optimized to achieve the minimization of the overall cost. The experiment on cloud-based DRT is conducted by using 6 real-world software programs with seeded faults and 3 simulative software programs. And the experiment results demonstrate that the cloud-based DRT strategy can achieve higher fault detection effectiveness than Round Robin strategy (RRS) and cloud-based RPT and has lower time and resource costs.

In conclusion, the fault detection effectiveness and efficiency of DRT are analyzed in this thesis, and the influence factors of the performance of DRT are investigated; D-DRT and the cloud-based DRT are proposed, and the performance of the proposed methods are evaluated and verified by extensive experimental studies. This thesis can provide significative guidance for the application of the testing strategies in practice.

    Research areas

  • Software cybernetics, Software testing, Dynamic random testing, Cloud testing, Resource allocation strategy