Assessing Conceptual and Empirical Contributions of Social Media Research Based on Knowledge Graph

Quality of research publications (“quality” for short) is an outmost concern by the scholarly community and beyond. However, how to measure quality has remained to be an open question. The prevailing practice is to use impact factor (based on citations) to measure quality. Although it is easy to obtain and analyze impact data, the measure is intended for popularity, not quality. A few scholars have attempted to measure quality based on expert reviews, without directly unpacking what core ingredients are inside quality. As such, quality has remained a black box. We propose an alternative approach (to assess contributions instead of quality) based on a new methodology (knowledge graph). Conceptually, we will focus on the newness of the scientific knowledge generated by a given publication (“knowledge” for short) over and above the existing knowledge. We further define a 3-dimensional typology, including empirical, operational, and conceptual, for knowledge measurement. The typology enables us to assess the newness of knowledge along the 3 dimensions in comparison with the relevant literature. If a publication reports a phenomenon that has already been observed and understood, the generated knowledge is a replication of “old wine”. If the publication provides a piece of new evidence for an existing hypothesis, or a new explanation for a known phenomenon, the relevant knowledge is a bottle of “mixed wine” between old and new. If the publication discovers a new phenomenon with a new explanation, the resulting knowledge is a “complete new wine”. In short, the knowledge-centric measure helps solve the long-standing dilemma between quality (i.e., an unmeasurable latent construct) and impact factor (a measurable but poor proxy for quality). Methodologically, we will develop and construct a knowledge graph (called “Social Media Knowledge Graph”, or SMKG for short) to assist the measurement and assessment of contributions of social media research. We will first identify and compile relevant studies of social media from Web of Science (estimated to be 30,000+) to form the source corpus. We will then employ a series of supervised and/or unsupervised machine learning models and algorithms to extract “knowledge ingredients” (i.e., theoretical concepts, their methodological or empirical attributes, and hypothesized and tested relationships among concepts) from the corpus. After applying co-reference resolution and concept disambiguation to the extracted knowledge ingredients, we will integrate them into a multilayer and dynamic knowledge base, in which theoretical concepts will be organized into different layers of abstraction and tagged with a time stamp based on the original date of publications. We will then create an ontology of social media research literature (i.e., SMKG) based on the knowledge base. The resulting SMKG will enable us to assess how much knowledge generated by each of the existing social media studies is “old wine” (mere replications), how much is “mixed wine” (new evidence for old hypotheses or new explanations for known phenomena), and how much is “brand new wine” (new phenomena with new explanations). Beyond the current project, the SMKG can be automatically updated in the future, which makes it possible to institute a real-time assessment system on the state of the arts of social media research. We also intend to extend, in the future, the underlying methodological framework to the assessment of other business/social research domains, after necessary modifications.