Biologically-inspired Reinforcement Learning and Concept-based Explanation for Multimodal Deepfake Detection in Financial Fraud Prevention

In his 2025 policy address, the Chief Executive of HK stress that the current government will facilitate wide-scale application and development of Artificial Intelligence (AI) across different sectors in the society to boost overall efficiency. However, malicious usage of AI has also led to widespread Deepfake attacks. The HK Business News reports that there is a 1,900% year-on-year increase in Deepfake-related fraud cases in the first quarter of 2025 in HK, and a 209% rise in synthetic identity document fraud in the city. Accordingly, the proposed GRF project aims to strategically aligned with the HKSAR Government’s long-term goal of promoting legitimate AI-enabled innovation in Hong Kong. Specifically, the proposed GRF project will develop a novel biologically-inspired deep reinforcement learning method and a generative concept-based explainable AI (XAI) method for multimodal Deepfake detection (e.g., video, audio, image, text) , and hence to prevent financial frauds such as fraudulent loan applications, AI-generated investment scams, forgery financial transactions, etc. Deepfake technology, which leverages Generative AI (GAI) models, has enabled the creation of highly convincing, fake multimodal contents to impersonate individuals, falsify financial documents, and manipulate communications, leading to substantial monetary losses. For instance, CNN reported that a finance worker in Hong Kong had mistakenly paid out HK$200 million after a video call with the firm’s “chief financial officer” deceptively generated by Deepfake technology on 4th Feb. 2024; Deepfake attacks bring significant risks to the business world, particularly for the financial industry. Existing static multimodal Deepfake detection methods are weak in combating rapidly evolving Deepfakes. Not only should the GAI-based detection models identify Deepfakes, but they should also proactively anticipate possible future attack patterns. Above all, existing detection models fail to explain to fraud investigators (e.g., loan managers) why certain identities or financial records are considered fraudulent based on explicit human understandable “concepts” to avoid risky business decisions (e.g., approving a loan for a fake client). Guided by the Design Science (DS) research methodology, the goal of the proposed GRF project is to address the aforementioned research gaps by designing a novel proactive, continual learning, and explainable multimodal Deepfake detection model which is underpinned by neuroplastic deep reinforcement learning and generative concept activation vector discovery and visualization methods. To ensure short/long term business and societal impacts, we plan for technology transfer of our research outcomes (e.g., a prototype system) through our listed industrial collaborator.