Tensor Criterion Based Neural Networks

Abstract

Tensor criterion is ubiquitously present across various domains, effectively guiding the design of neural networks for modeling multi-dimensional relationships and complex interactions, such as user-item-context interactions in recommender systems, spatial-temporal correlations in traffic prediction, and multi-modal features in computer vision. The increasing complexity of real-world data analysis and processing has driven numerous scholars to explore tensor criterion-based methods. Among these approaches, tensor criterion-based neural networks stand out for their ability to naturally model and process multi-dimensional data, establishing principled optimization objectives that support more accurate predictions and decision-making, which have become a research hotspot in recent years.

By viewing neural networks through the lens of tensor criteria and mathematics, we can obtain deeper insights into their inherent structure and optimization principles, leading to more principled architectural designs and efficient learning algorithms. This thesis investigates the fundamental connections between tensor criteria and neural networks, designing novel architectures that leverage tensor principles to optimize both computational efficiency and model effectiveness. The developed methods demonstrate the power of tensor criterion-based thinking across various practical applications. The main research content and innovations of this thesis are as follows:

Firstly, this thesis identifies key challenges in current neural network architectures from a tensor criterion perspective: the difficulty in establishing proper optimization objectives for high-dimensional missing data, the limitations in formulating criteria for complex feature interactions, the privacy-preserving criterion in multi-source data modeling, and the efficiency criterion in model adaptation. To address these challenges, the thesis proposes several innovative tensor criterion-based solutions. The ConFact framework is developed with carefully designed tensor criteria for efficient missing data completion, incorporating dimension-specific convolutions and interpretable matrix factorization principles. Furthermore, these methods demonstrate superior performance in both accuracy and computational efficiency compared to existing approaches.

Secondly, this thesis presents THNN, a novel tensorized hypergraph neural network with a principled tensor criterion that fundamentally changes how high-order feature interactions are modeled. Unlike traditional methods that rely on first-order approximations, THNN utilizes adjacency tensors to directly formulate optimization objectives for complex feature relationships. This innovation enables more accurate feature fusion and achieves significant improvements in tasks such as hypergraph-based 3D visual object classification.

Thirdly, the thesis introduces FLEST, a federated tensor factorization framework with privacy-preserving tensor criteria that addresses the critical challenge of privacy-preserving knowledge sharing across distributed sources. By innovatively decomposing embedding matrices into dictionary and loading components based on carefully designed tensor objectives, FLEST enables effective knowledge graph completion while maintaining data privacy under federated settings.

Fourthly, this thesis proposes Tenta-LoRA, an efficient model adaptation method that leverages tensor criteria to enhance the traditional Low-Rank Adaptation approach. This innovation establishes principled optimization objectives that significantly reduce model parameters while maintaining performance, providing a practical solution for resource-constrained applications.

Experimental results demonstrate that these tensor criterion-based approaches consistently outperform existing methods across various tasks. The ConFact achieves superior accuracy in matrix completion tasks through well-designed tensor criteria. THNN shows remarkable improvements in hypergraph-based 3D object classification with its novel tensor optimization objectives. FLEST effectively balances privacy and utility in federated scenarios through privacy-aware tensor criteria. Tenta-LoRA achieves efficient model adaptation with minimal performance loss through tensor-based optimization principles.

This thesis concludes by summarizing its contributions and discussing future research directions. Potential extensions include developing more sophisticated tensor criteria for emerging applications, investigating theoretical foundations of tensor criterion-based architectures, and exploring novel optimization objectives in quantum computing and molecular modeling. These innovations not only advance specific application domains but also provide new perspectives for designing neural networks through tensor criterion principles.

Date of Award	5 Sept 2025
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Xiangyu ZHAO (Supervisor), Junhui WANG (Supervisor) & Ruocheng GUO (Co-supervisor)