Term Weighting Schemes for Question Categorization

Term weighting has been proved to be an effective way to improve the performance of text categorization. Very recently, with the development of user-interactive question answering or community question answering, there has emerged a need to accurately categorize questions into predefined categories. However, as a question is usually a piece of short text, can the existing term weighting methods work consistently on question categorization as they do on text categorization? The answer is not clear since to the best of our knowledge, we have not seen any related work to study this problem despite of its significance. In this project we will investigate the popular unsupervised and supervised term weighting methods for the question categorization task. At the same time, in view of the specificities of questions, we propose three new supervised term weighting methods, namely, qf*icf, iqf*qf*icf and vrf. Comparisons of them with existing unsupervised and supervised term weighting methods will be made through a series of experiments on large-scale question collections from Yahoo! Answers. The project can lead to better text representation of questions in User- Interactive QA and to improve question categorization. Furthermore, the findings of this project can be applied to other short text snippets, which have similar characteristics to questions.