Multimodal graph learning based on 3D Haar semi-tight framelet for student engagement prediction

With the increasing availability of multimodal educational data, there is a growing need to effectively integrate and exploit multiple data sources to enhance student engagement prediction accuracy. In this work, we propose a framework that combines multimodal data, including visual, textual and acoustic modalities that reflect the students’ personalities, their demographic information, their learning behavior and attention, with graph learning techniques. Specifically, 3D Haar semi-tight framelet transforms are developed to capture the inter-modal relationships and model the complex interactions within the multimodal data. Subsequently, we introduce a novel module for adaptive graph structure learning based on the spectrum of multimodal data, which takes into consideration the distinct contributions of low-pass and high-pass framelet coefficients by adaptively weighing their impact. By addressing a standard semi-supervised node classification problem, we successfully achieve the objective of student engagement prediction. The experiment evaluations on a real-world educational dataset demonstrate the effectiveness of the proposed approach, achieving superior performance compared to state-of-the-art methods. Our experimental studies demonstrate the importance of multimodal graph learning in accurately predicting student engagement and its potential to enhance educational outcomes. © 2024 Elsevier B.V. All rights reserved.