Assessment in Sports Videos: A Deep Learning Approach with 3D Human Posture Integration

Action Quality Assessment (AQA) is emerging as a significant tool in sports analytics, aiming to provide more objective evaluations of athletic performances. This research aims to enhance the methodologies of Action Quality Assessment (AQA) by using both skeletal and visual data in sport videos. Contemporary AQA techniques primarily utilize either pose-based or vision-based inputs. However, pure vision-based methods often falter when exposed to complex environmental variables and may miss out on crucial aspects of human mobility. Pure pose methods ignore details in the scene, such as the splash of diving, and traditional 2D key points cannot fully encapsulate the intricacies of human motion. To address these shortcomings, we propose to implement 3D skeletal-data- based methods and combine the RGB videos. We plan to extract 3D human posture information from videos to better gauge action quality. This involves a modular two-stream network that combines the 3D pose with appearance features from RGB videos. Moreover, we will look into another related problem that is to enhance AQA for long sports videos. Long videos often have complex motion sequences which necessitate a thorough understanding of both local and global spatio-temporal relationships. We propose a systematic segmentation approach for long videos to divide them into multiple atomic actions. Besides, we will introduce an attention mechanism to prioritize specific sections of the video that carry more weight in terms of assessment. Afterwards, we plan to explore the research issue for transferring AQA to new domains. Recognizing that some aspects of action quality are consistent across various sports, we will investigate how the principles of AQA can be extended to the general public, for instance in evaluating exercise form. We propose to adopt a multi-action based AQA model and subsequently fine-tune it using data from public exercise videos. Our research, backed by extensive evaluations using diverse datasets, holds the potential to change the way we understand and evaluate sports and exercise videos, promising more accurate, comprehensive, and adaptable action quality assessments. Our work can assist athletes to identify ways to improve their performance. Besides, it can also help the general public to have a better understanding of sports activities, since many people are excited to watch professional sports especially in major events such as Summer and Winter Olympics or Asian Games.