Interactive Video Colorization with Memory Units and Feature Correction

Personalized colorization of grayscale videos has extensive application potential in consumer electronics, but it is a challenging problem due to occlusions and error accumulation. Recently, user-guided image colorization has been intensively studied with significant progress. However, directly applying image algorithms to video colorization will introduce serious consistency problems. Existing methods solve this problem by improving loss functions, post-processing, etc., but they over-rely on inaccurate optical flow, resulting in poor performance. In this paper, we improve the temporal consistency through feature correction and propose an interactive video colorization network called MCVCNet. To this end, we propose maintaining a set of memory units that record the colorization state and guide subsequent frames. Subsequently, a correction module is proposed, which can modulate the semantic features of the current grayscale frame using the memory units to eliminate errors caused by network deepening and inaccurate optical flow. The independence of the memory units and the correction module makes it easy to apply an image colorization network to interactive video colorization tasks. Furthermore, we build a large-scale dataset specifically for video colorization. Experiments demonstrate that our method outperforms other methods in producing high-fidelity videos and maintaining temporal consistency both qualitatively and quantitatively.

© 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.