Shared Multimodal Data Representation Learning for Multi-sourced Event Detection

The Internet technologies have been the primary force behind the popularity of Webnews media (e.g., BBC News, ABC New, NBC News, CNN News, etc.), and the fast rise ofsocial media sites (e.g., Flickr, Facebook, Twitter, etc.). The popularity of the Web newsmedia and, since Web2.0 era, social media services, has resulted in huge amounts of(professional) journalist-reported data and user-generated data streams on the Internet.Such massive data resources serve as a treasure tank for people to explore the real-worldconcepts, express personal/public views, and share or report events, thereby“sensing the world” timely. As real-world events are closely related to people’s lives, theyhave received increasingly more attention in recent years. Multi-sourced EventDetection (MED) aims to mine and monitor various real-world events, in an efficientmanner, from both the user-shared social media data and the journalist-reported newsmedia data, so as to provide comprehensive and timely views of the events.However, MED is deemed as challenging, due to the following two primary reasons. First,the data streams are often multimedia ones which may possess multiple heterogeneousfeatures like text, images, meta-data (e.g., timestamp-taken, GPS-tags, tags, useridentity, etc.). Such multimodal data, while providing necessary and complementaryinformation about the events, are generally difficult for traditional machine learningmodels to fully exploit. Second, the raw data contributed by different people rangingfrom ordinary users to journalists can be diverse and rather low in discriminative powerdue to data variations. For example, image photos may be taken from multipleviewpoints using different devices at different time, resulting in varying viewpoints,resolutions, illumination conditions, changing backgrounds, etc. Likewise, textualnews/descriptions may be written by different people with various levels ofprofessionalism from different standpoints, etc.In this project, we aim to: 1) address the issues of heterogeneity and low discriminativepower of raw data, through devising a shared multimodal data representation learningmodel by fully utilizing the multi-sourced multimodal data; 2) improve the effectivenessand efficiency of event detection by devising hybrid clustering models and class-wiseresidual models; and 3) validate the effectiveness of our proposed modals upon real-worlddata. An experimental prototype system is to be developed to manifest the processof MED, and extensive testing will be conducted by using real-world data to evaluateour proposed research.