Task-aware Visual Mesh Saliency

Description

The concept of visual saliency or where people look is interesting as we all continuously pay visual attention to the world that we live in. In computer graphics, this concept has been explored for 3D shapes to consider where people look on 3D shape surfaces. Furthermore, the idea of task-based saliency has been studied for 2D images to consider whether the viewer task affects where people look in images. However, task-aware saliency for 3D shapes has not been explored and we propose to fill this gap. Our overall research challenge is to develop a computational understanding of task-aware visual mesh saliency, and our central hypothesis is that the viewer task will affect where people look on 3D shapes. This is an important problem as there are potential applications in advertising and 3D content creation. The proposed work builds on the PI’s previous projects in 3D shape perception and in the use of machine learning methods to learn perceptual properties of shapes.  We will take a user-oriented approach, and directly collect perceptual data of where humans pay attention to for different shapes and tasks to quantify our saliency concept. Our work will collect the first task-aware mesh saliency dataset. We propose a data collection process that will comprehensively explore the experimental design space. In particular, we come up with as many viewer tasks, 3D shape representations, and saliency collection methods as we can and consider all possible combinations of them. We then perform both qualitative and quantitative analysis of the data to understand why the viewer’s attention for each shape may be different across different tasks. We then explore various approaches to learn functions that take as input a 3D point (or shape) and a viewer task, and compute as output the task-aware mesh saliency score (or map). In contrast to previous work, we consider the task-aware aspect for 3D shapes and the temporal aspects of the gaze data. We propose to develop new task-aware neural network architectures including one that adapts the computation time in different time steps of the gaze prediction. Finally, we will develop several task-aware 3D shape applications. These include exploring the reverse problem of predicting the viewer’s task given the gaze data for understanding human behavior, selecting mesh viewpoint with the task in mind for more effective advertising, and revisiting 3D shape problems from the task perspective for better content creation in games and films.