Projects per year
Abstract
While language models have become impactful in many real-world applications, video generation remains largely confined to entertainment. Motivated by video's inherent capacity to demonstrate physical-world information that is difficult to convey through language alone (eg, imagine teaching someone to tie a tie using only text), we identify an underutilized opportunity to extend video as a new answer modality for Next-Event Prediction (NEP), formalized as Video-Next-Event Prediction (VNEP). While the established NEP task takes a video with a procedural or predictive question as input to predict the next event in text, VNEP requires dynamic video responses. This shift from telling to showing unlocks more intuitive and customized answers for procedural learning and creative exploration. However, this task remains challenging for existing models, as it demands an understanding of multimodal input, instruction-conditioned reasoning, and the generation of video with visual and semantic consistency. To address this, we introduce VANS, a model that leverages reinforcement learning to align a Vision-Language Model (VLM) with a Video Diffusion Model (VDM) for VNEP. The core of VANS is our proposed Joint-GRPO that orchestrates the VLM and VDM to function as a unit. Driven by a shared reward on their respective output, it optimizes the VLM to produce captions that are both accurate and friendly to visualize, while guiding the VDM to generate videos that are faithful to these captions and the input visual context. To enable this learning, we craft VANS-Data-100K, a dedicated dataset for the VNEP task. Experiments
on procedural and predictive benchmarks demonstrate that
VANS achieves state-of-the-art performance in both video
event prediction and visualization. Codes are released in
https://github.com/KlingTeam/VANS.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) |
| Pages | 38915-38925 |
| Publication status | Published - Jun 2026 |
| Event | 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2026) - Colorado Convention Center, Denver, United States Duration: 3 Jun 2026 → 7 Jun 2026 https://cvpr.thecvf.com/ |
Conference
| Conference | 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2026) |
|---|---|
| Place | United States |
| City | Denver |
| Period | 3/06/26 → 7/06/26 |
| Internet address |
Bibliographical note
Research Unit(s) information for this publication is provided by the author(s) concerned.Funding
This work was supported by Kuaishou Technology and partially supported by the Innovation and Technology Fund (ITF) of the Innovation and Technology Commission (ITC) of the Hong Kong Special Administrative Region (HKSAR) Government [Project No. ITS/269/24FP].
Fingerprint
Dive into the research topics of 'Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO'. Together they form a unique fingerprint.Projects
- 1 Active
-
ITF: Next-Generation Product Placement: Combining CG and AI Technologies for Intelligent Advertising
LIAO, J. (Principal Investigator / Project Coordinator), HUANG, Z. (Co-Investigator), WANG, C. (Co-Investigator) & Zhang, Z. (Co-Investigator)
1/05/26 → …
Project: Research
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver