Research on Recommendation Systems based on Multimodality Content and User Inherent Profiling

Recommender systems are widely applied in e-commerce, social media, and content platforms to guide consumers in browsing or purchasing various items, such as products, short videos, movies, and restaurant services, etc. In recent years, multimodal content marketing has rapidly emerged, for instance, image-text posts and short video promotions, where platforms leverage multimodal data such as images, textual descriptions, and user reviews to provide more vivid presentations and interactive experiences. Multimodal content not only enhances the user experience but also drives technical upgrades in recommender systems. Facing the diversity of items and the complexity of user demands, recommender systems have evolved from traditional collaborative filtering and unimodal content matching toward multimodal fusion and user latent profiling. User latent profiles, such as long-term interests or price sensitivity, are usually unobservable. In recent years, the rapid development of Large Language Models (LLMs) has provided new opportunities for recommender systems. With extensive world knowledge and powerful semantic understanding capabilities, LLMs can assist recommender systems in analyzing user profiles from multimodal content. However, with large-scale user interaction logs and multimodal data, how to effectively integrate user histories, reviews, images, and textual descriptions, while leveraging the semantic understanding capabilities of LLMs to infer latent profiles and preferences, has become a pressing challenge for improving recommendation accuracy and user engagement.

This thesis focuses on recommender system optimization in the context of multimodal content marketing, and fuses multimodal content features, models user-item interactions, and infers users' latent profiles. The effectiveness of the proposed approaches is validated on real-world datasets. The thesis consists of four parts:

Firstly, the thesis investigates the recommendation problem with multimodal content, such as image-text posts, and proposes a recommender system considering multimodal contrastive representations and dynamic preference, which can accurately extract user preferences from multimodal content. In practice, user preferences are jointly influenced by multimodal information such as an item’s textual and visual content as well as item ID. Accurately understanding user interests based on multimodal content has therefore become a major challenge for recommender systems. Current approaches typically model user preferences by encoding visual and textual features, while overlooking users’ dynamic preferences across different modalities, which undermines recommendation accuracy. To address this, the thesis integrates multimodal content (including visual images, text descriptions, and item IDs) into the recommendation system and designs a recommendation system considering multimodal contrastive representation and dynamic preferences. This thesis introduces a contrastive information maximization loss, constructs multimodal prototypes and noise-contrastive sequences, and trains multimodal contrastive representations that capture users’ varying attention to different modalities. This method strengthens correlations between cross-modal information, significantly improving accuracy and robustness. The study shows that incorporating multimodal content into user preference modeling significantly improves prediction accuracy and enhances the model’s interpretability of user behavior. The proposed method consistently outperforms all baseline models in terms of Hit Rate (HR) and Normalized Discounted Cumulative Gain (NDCG) across three real-world datasets. Extensive experiments reveal that multimodal contrastive representations play a crucial role in capturing users’ preferences for visual and textual modalities, and further validate the effectiveness and interpretability, offering insights for future multimodal recommendation systems.

Secondly, this thesis examines the recommendation problem with user interest drift and proposes a recommender system based on Graph Convolutional Networks (GCNs) for multi-scale interest modeling and multimodal dynamic fusion. In practice, users’ interest preferences are not static but continuously evolve over time, influenced by factors such as temporal dynamics and trending content. For multimodal recommendation systems, it is essential to jointly capture both the temporal dynamics of user interests and their multimodal preference patterns. However, due to the dynamic drift of user interests, existing systems struggle to uncover implicit temporal patterns embedded within multimodal features. Most current approaches rely on static multimodal feature fusion (e.g., averaging or simple concatenation), which fails to adaptively capture modality-specific preference variations across users and lacks explicit modeling of interests at multiple temporal scales. This thesis investigates the dynamic correlations between users’ evolving interest preferences across different time horizons and multimodal item features, and proposes a recommendation system based on GCNs for multi-scale interest modeling and multimodal dynamic fusion. The system leverages pre-trained models to extract multimodal embeddings from item images, textual descriptions, and IDs. It innovatively integrates a multimodal dynamic fusion module with long-, medium-, and short-term interest modeling to predict the next-item. The multimodal dynamic fusion module adaptively captures users’ shifting preferences across modalities by computing dynamic attention scores over multimodal features. Meanwhile, a multi-scale GCN encoder is designed to jointly model temporal behavior patterns at long-, medium-, and short-term horizons, thereby enhancing the accuracy of user behavior representation. Extensive experiments on three real-world datasets, with ablation analyses on multi-scale interests, demonstrate that the proposed method significantly outperforms state-of-the-art baselines in terms of HR and NDCG, confirming the value of modeling interest drift across long-, medium-, and short-term horizons.

Thirdly, this thesis addresses the recommendation problem with user latent profiles and proposes a recommender system that integrates LLMs prompt tuning with latent profile inference, enabling personalized recommendations. In recommender systems lacking structured user data, extracting user profiles from historical records is an important technique for analyzing user behaviors and profiles. However, current methods mainly focus on explicit preferences in user interactions, which is difficult to infer latent profiles (e.g., family background, long-term interests), thereby limiting the precision of user profiles inference. Meanwhile, although LLMs possess strong semantic understanding capabilities, their effectiveness is constrained by issues such as unstandardized instructions, modality gaps, high inference latency, and noisy textual outputs. To address these challenges, this thesis incorporates user interactions and proposes a recommender system based on LLMs prompt tuning and latent profile inference. Specifically, it employs soft tokens to efficiently infer latent profiles, encodes them into quantized IDs through vector quantization, and feeds these IDs into downstream recommenders to predict user interaction probabilities, thus improving both efficiency and accuracy in collaborative signal extraction. Experiments on real-world datasets demonstrate that the proposed method significantly outperforms state-of-the-art baselines, with latent profile variables effectively capturing user features. Large-scale industrial experiments further confirm the model’s superior inference efficiency, achieving substantial speedup while maintaining predictive accuracy. The proposed method has been deployed in Huawei AppGallery’s content card recommendation system, where an online A/B test shows a significant increase in business benefits. 

Finally, this thesis investigates the recommendation problem considering user price sensitivity profile, and proposes a recommender system that integrates LLMs agents with a user price-sensitivity memory to infer user preferences and price-sensitivity features. User reviews not only reflect individual interests but also implicitly reveal their price sensitivity, an essential latent preference that critically influences purchase decisions but cannot be directly observed from clicks or browsing behaviors. Leveraging LLMs’ strong semantic understanding and reasoning capabilities, LLMs can extract user intent from reviews and infer both user interests and price sensitivity profiles. To this end, this thesis proposes a recommender system that combines LLMs agents with a user price-sensitivity memory, further enhanced by a rotation vector quantization for efficient user profile clustering. Specifically, the agent performs semantic reasoning over user reviews to extract preference and price-sensitivity profiles, after which a memory-update agent dynamically updates the user profile to model the evolution of interests and consumption psychology. To improve the efficiency of downstream recommenders in utilizing user profiles, a rotation vector quantization approach is designed to map semantic vectors encoded by pretrained models into discrete user profile IDs. These IDs are then used as augmented input attributes for downstream recommenders, thereby enhancing recommendation efficiency. The proposed method is validated on real-world review datasets, with a newly designed evaluation metric, price-aware hit rate, to assess whether the system can recommend items within users’ acceptable price ranges, thereby measuring the accuracy of the price-sensitivity profile. Experimental results demonstrate that the proposed system significantly outperforms existing methods across multiple evaluation metrics while exhibiting strong generalization ability.

Moreover, beyond pursuing accuracy gains, the proposed systems pay attention to computational efficiency and deployability. The thesis improves multimodal representation and latent profile inference through efficient contrastive learning, adaptive multimodal fusion, vector quantization, and memory-enhanced profile encoding for practical recommender systems. These contributions deliver a multimodal and inherent profiles-aware recommendation framework, offering practical guidance for real-world deployment in modern content marketing and retail platforms.