A Short Text Similarity Assessment Method Based on Probabilistic Topics and Its Application to Question Categorization

Short text refers to text snippets with just a few sentences or even a few words. Currently, effective short text modelling method is in urgent need as more and more new applications related to short text are developed, including short message services, instant messenger, BBS, forum, and question answering. Existing techniques (e.g., TF-IDF or synonym-based) developed for long text documents cannot be effectively applied to these applications since much less common words or synonyms can be found in similar short text. Therefore, how to accurately discover the actual relevance between short texts is still a great challenge.This project will investigate more effective methods for similarity calculation specifically for short text. The researchers will focus on how to model (a collection of) short text snippets and measure their similarities. One of the greatest, maybe the major, obstacles of discovering the real relevance between two short texts is that short texts contain a few uncommon words while the total words in each texts are limited. Based on this sense, they aim to mine the relevance of short texts from the uncommon words with a statistical method. Specifically, the researchers assume that the similarity between two short texts is relevant to two components, one is the common words between them and the other component is their distinguishing terms. This method aims to mine the implicit relevance between the distinguishing terms by comparing them with several third-party agencies. In this project, the relevance between the distinguishing terms is discovered by examining their probabilities in a series of probabilistic topics, extracted by the Gibbs sampling algorithm. Finally, the similarity between two short texts can be calculated based on their common words and the relevance of their distinguishing terms.The researchers will compare our methods with existing similarity methods in terms of their performance in certain applications, including question categorization.