A Deep Learning Pipeline to Restore Images or Videos with Unknown and Mixed Defects

Image restoration is a classical but still active research topic in computer vision and image processing because it is fundamental to many applications such as photography, surveillance, and medical imaging. Image restoration is also a challenging ill-posed problem due to the information loss during the corruption procedure. Traditional methods define some priors to tackle each subproblem including denoising, deblurring, super resolution, inpainting, and colorization. Recently, with the success of deep learning, many different neural networks have been proposed to solve restoration tasks by utilizing large-scale training data. These networks are still designed for one specific task with a known corruption model. However, real-world corrupted images or videos, such as old photographs, historic artworks, or vintage movies, often simultaneously suffer from multiple artifacts caused by unknown and mixed corruption functions, which cannot be successfully restored using existing single-task methods. Therefore, we propose novel techniques for automatically restoring images from unknown and mixed defects and then extend them to videos. To achieve this, we have several targets. The first one is to design a uniform deep neural network structure that can be applied to different single-restoration tasks. This network is constructed by a pyramid of subnetworks with residual blocks, which progressively recover the target image residue in a coarse-to-fine way to get better quality. Our second target is to make this uniform network support joint training of multiple restoration tasks with one model; thus, different tasks can benefit from each other. This is realized by representing each task with a filter bank plugged into the shared uniform network structure. With the above two basics, our central target to restore images suffering from unknown and mixed real-world corruptions can be achieved by learning the combination of pretrained basic restoration tasks represented by filter banks in our network in an unsupervised manner. Moreover, we plan to add exemplar guidance into our networks, which enables simply feeding different references to customize results; and extend our method from image to video by imposing temporal constraints to generate temporally coherent results. The project is built on our previous works on deep-learning-based image/video processing and generation. Our preliminary experiments have already demonstrated both the feasibility and the quality of some building blocks in the proposed methodologies. Our proposal is the first attempt to tackle multiple, mixed, and unknown corruptions in images and videos, which will bring the state-of-the-art restoration techniques to the next level in solving real-world problems.