Modality-aligned fine-tuning of large models for stock prediction

There is a significant gap between the outstanding performance of Large Language Models (LLMs) in language tasks and their direct application in temporal financial prediction. Traditional fine-tuning methods struggle to overcome challenges such as low signal-to-noise ratio in numerical time-series data and complex dynamic patterns. To address these challenges, we propose an efficient adaptation framework LLaMA-TTPT for frozen LLMs. LLaMA-TTPT addresses the limitations of utilization efficiency and robustness with multi-modal prompt, cross attention and layer-wise hierarchical prompt propagation. The core innovation of this paper is implicitly mapping temporal data into the semantic space of pretrained LLMs. Firstly, we proposed a temporal-text prompt fine-tuning mechanism, which employes trainable numerical encoders and text prompts to convert stock price sequences into LLM-aligned multi-modal inputs. This enhances cross-modal consistency between numerical features and semantic descriptions. Secondly, we adopt cross attention mechanism to fuse dual-modal embeddings. Finally, we propose a temporal-aware hierarchical soft prompt propagation structure. It combines local dynamic prompt with global trend features extracted through LLMs' self-attention, enabling collaborative modeling of long-short term dependencies. Experiments on India Nifty 50 Stock Market dataset demonstrate that LLaMA-TTPT outperforms mainstream temporal models. Ablation analyses further validate the effectiveness of the proposed methods. © 2025 Society of Photo-Optical Instrumentation Engineers (SPIE).