A statistical method for middleware system architecture evaluation

The architecture of complex software systems is a collection of decisions that are very expensive to change. This makes effective software architecture evaluation methods essential in today's system development for mission critical systems. We have previously developed MEMS for evaluating middleware architectures, which provides an effective assessment of important quality attributes and their characterizations. To provide additional quantitative assessments on the overall system performance using actual runtime data, we employed a set of statistical procedures in this work. Our proposed assessment procedures comprises a standard sensitivity analysis procedure that utilizes leverage statistics to identify and remove influential data points, and an estimator for evaluating system stability and a metric for evaluating system load capacity. Experiments were conducted using real runtime datasets. Results show that our procedures effectively identified and isolated abnormal data points, and provided valuable statistics to show system stability. Our approach thus provides a sound statistical basis to support software architecture evaluation. © 2010 IEEE.