Assessing the representativeness of open source projects in empirical software engineering studies

Research output: Chapters, Conference Papers, Creative and Literary Works (RGC: 12, 32, 41, 45)32_Refereed conference paper (with ISBN/ISSN)

2 Scopus Citations
View graph of relations

Author(s)

Detail(s)

Original languageEnglish
Title of host publicationProceedings - Asia-Pacific Software Engineering Conference, APSEC
PublisherIEEE Computer Society
Pages808-817
Volume1
ISBN (Print)9780769549224
Publication statusPublished - 2012
Externally publishedYes

Publication series

Name
Volume1
ISSN (Print)1530-1362

Conference

Title19th Asia-Pacific Software Engineering Conference, APSEC 2012
PlaceChina
CityHong Kong
Period4 - 7 December 2012

Abstract

BACKGROUND: Software engineering researchers have carried out many empirical studies on open source software (OSS) projects to understand the OSS phenomenon, and to develop better software engineering techniques. Many of these studies typically use only a few successful projects as study subjects. Recently, these studies have received criticisms and challenges on their representativeness on OSS projects. AIM: First, we aim to examine to what extent data extracted from successful projects are different from data extracted from the majority. If data extracted from successful projects are quite different from data extracted from the majority, approaches that are effective on successful projects may not be effective in general. Second, we aim to examine whether successful OSS projects are representative to the whole population of OSS. If they are not, conclusions that are drawn from only successful projects may reflect the OSS phenomenon partially. METHODOLOGY: We analyzed 11, 684 OSS projects that are hosted on Source Forge. When researchers select subjects, they typically select successful projects that are attractive to both users and developers. Considering this preference, we clustered these projects into four categories based their attractiveness to users and developers. Here, we use the K-means clustering technique to produce combined result. Furthermore, we selected eight indicators that are used in many existing studies (e.g., team sizes), and compared indicators that are extracted from different categories to investigate to what degree they are different. RESULT: For the first research aim, the result shows that 66.1% projects are under developing projects, 14.7% projects are user-preference projects, 14.2% projects are developer-preference projects, and only 5.0% projects are considered successful. For the second research aim, the result shows that all the eight analyzed indicators are highly unbalanced with the gamma distribution. Furthermore, the result reveals that users and developers of Source Forge have different perceptions on the development status defined by Source Forge. CONCLUSION: We conclude that successful projects are not representative to the whole population of OSS, and data extracted from successful projects are quite different from data extracted from the majority. The result implies that conclusions drawn from only a few successful projects may be challenged. This work is important to allow researchers to refine conclusions of existing studies, and to better understand and to carefully select OSS project subjects for their future empirical experiments. © 2012 IEEE.

Research Area(s)

  • Empirical Studies, Open source software

Citation Format(s)

Assessing the representativeness of open source projects in empirical software engineering studies. / Zhong, Hao; Yang, Ye; Keung, Jacky.

Proceedings - Asia-Pacific Software Engineering Conference, APSEC. Vol. 1 IEEE Computer Society, 2012. p. 808-817 6462750.

Research output: Chapters, Conference Papers, Creative and Literary Works (RGC: 12, 32, 41, 45)32_Refereed conference paper (with ISBN/ISSN)