Spatiotemporal Super-Resolution with Cross-Task Consistency and Its Semi-supervised Extension

Research output: Chapters, Conference Papers, Creative and Literary Works (RGC: 12, 32, 41, 45)32_Refereed conference paper (with ISBN/ISSN)

View graph of relations

Author(s)

Detail(s)

Original languageEnglish
Title of host publicationProceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
EditorsChristian Bessiere
PublisherInternational Joint Conferences on Artificial Intelligence
Pages615-622
ISBN (Electronic)978-0-9992411-6-5
Publication statusPublished - Jan 2021

Conference

Title29th International Joint Conference on Artificial Intelligence (IJCAI 2020)
LocationOnline
PlaceJapan
CityYokohama
Period7 - 15 January 2021

Abstract

Spatiotemporal super-resolution (SR) aims to upscale both the spatial and temporal dimensions of input videos, and produces videos with higher frame resolutions and rates. It involves two essential sub-tasks: spatial SR and temporal SR. We design a two-stream network for spatiotemporal SR in this work. One stream contains a temporal SR module followed by a spatial SR module, while the other stream has the same two modules in the reverse order. Based on the interchangeability of performing the two sub-tasks, the two network streams are supposed to produce consistent spatiotemporal SR results. Thus, we present a cross-stream consistency to enforce the similarity between the outputs of the two streams. In this way, the training of the two streams is correlated, which allows the two SR modules to share their supervisory signals and improve each other. In addition, the proposed cross-stream consistency does not consume labeled training data and can guide network training in an unsupervised manner. We leverage this property to carry out semi-supervised spatiotemporal SR. It turns out that our method makes the most of training data, and can derive an effective model with few high-resolution and high-frame-rate videos, achieving the state-of-the-art performance. The source code of this work is available at https://hankweb.github.io/STSRwithCrossTask/.

Citation Format(s)

Spatiotemporal Super-Resolution with Cross-Task Consistency and Its Semi-supervised Extension. / Lin, Han-Yi; Hsiu, Pi-Cheng; Kuo, Tei-Wei; Lin, Yen-Yu.

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. ed. / Christian Bessiere. International Joint Conferences on Artificial Intelligence, 2021. p. 615-622.

Research output: Chapters, Conference Papers, Creative and Literary Works (RGC: 12, 32, 41, 45)32_Refereed conference paper (with ISBN/ISSN)