Can earnings conference calls tell more lies? A contrastive multimodal dialogue network for advanced financial statement fraud detection

Financial statement frauds by listed firms pose significant challenges to public investors and jeopardize the stability of financial markets. Previous studies have identified deceptive verbal and vocal cues from earnings conference calls as indicators of financial statement fraud. However, these studies only extracted managers' verbal and vocal cues separately over the entire call, neglecting the utterance-level fusion between verbal and vocal cues as well as the multi-turn interaction between analysts and managers. To fill this gap, we develop a novel end-to-end contrastive multimodal dialogue network (CMMD) that considers both verbal-vocal fusion and multi-role interactions to uncover hidden deceptive cues in earnings conference calls. The proposed model comprises two core modules, namely, the Multimodal Fusion Learning module and the Dialogue Interaction Learning module. Building on Vrij's verbal-nonverbal complementary mechanisms in deception detection, the designed Multimodal Fusion Learning employs contrastive learning to align verbal and vocal cues and a co-attention mechanism to learn cross-modal interaction. Inspired by the Interpersonal Deception Theory that emphasizes the dynamic interaction process between deceivers and targets, the Dialogue Interaction Learning utilizes a dialogue-aware co-attention mechanism to model multi-turn analyst-manager interaction and uses contrastive learning to improve dialogue representations. Our extensive empirical results show that CMMD achieves 8.64 % improvement in detecting fraudulent cases compared to the best baseline model. As such, our study advances the research frontier in fraud detection and contributes an innovative IT artifact in practice. © 2024 Published by Elsevier B.V.