New trend on chemical structure representation learning in toxicology: In reviews of machine learning model methodology

Computer-assisted virtual screening using structure-activity relationship (QSAR) models is a surrogate method for reducing the need for costly animal experiments. However, traditional QSAR models face significant challenges, such as the ‘activity cliff’ phenomenon and small datasets, which limit their ability to generalize and predict toxicity. This review examines transistion of digital encodings form in molecules and its corresponding models, introducing from molecule descriptors to three advanced types of molecular representations based on deep learning techniques. We highlight the importance of deep learning models that can not only capture molecular similarity in chemical space to address the ‘activity cliff’ problem but also improve model performance through feature fusion. As alternative solutions to reduce reliance on feature engineering potentially, graph neural network, convolutional neural network and large lanuage model and their related training paradigm such as transfer learning could give another opportunity for toxicity model setting in terms of data insuffient dealing etc. This work could help potential deep learning modelers to build robust model, setting the stage for groundbreaking advancements in further development and application of toxicity prediction models. © 2025 Taylor & Francis Group, LLC.