Scaling up analogy-based software effort estimation: a comparison of multiple hadoop implementation schemes

Analogy-based estimation (ABE) is one of the most time consuming and compute intensive method in software development effort estimation. Optimizing ABE has been a dilemma because simplifying the procedure can reduce the estimation performance, while increasing the procedure complexity with more sophisticated theory may sacrifice an advantage of the unlimited scalability for a large data input. Motivated by an emergence of cloud computing technology in software applications, in this study we present 3 different implementation schemes based on Hadoop MapReduce to optimize the ABE process across multiple computing instances in the cloud-computing environment. We experimentally compared the 3 MapReduce implementation schemes in contrast with our previously proposed GPGPU approach (named ABE-CUDA) over 8 high-performance Amazon EC2 instances. Results present that the Hadoop solution can provide more computational resources that can extend the scalability of the ABE process. We recommend adoption of 2 different Hadoop implementations (Hadoop streaming and RHadoop) for accelerating the computation specifically for compute-intensive software engineering related tasks.