Sketch-R2CNN : An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

View graph of relations

Author(s)

  • Lei Li
  • Changqing Zou
  • Youyi Zheng
  • Qingkun Su
  • Chiew-Lan Tai

Related Research Unit(s)

Detail(s)

Original languageEnglish
Article number9068451
Pages (from-to)3745-3754
Journal / PublicationIEEE Transactions on Visualization and Computer Graphics
Volume27
Issue number9
Online published15 Apr 2020
Publication statusPublished - 1 Sep 2021

Abstract

Sketches in existing large-scale datasets like the recent QuickDraw collection are often stored in a vector format, with strokes consisting of sequentially sampled points. However, most existing sketch recognition methods rasterize vector sketches as binary images and then adopt image classification techniques. In this paper, we propose a novel end-to-end single-branch network architecture RNN-Rasterization-CNN (Sketch-R2CNN for short) to fully leverage the vector format of sketches for recognition. Sketch-R2CNN takes a vector sketch as input and uses an RNN for extracting per-point features in the vector space. We then develop a neural line rasterization module to convert the vector sketch and the per-point features to multi-channel point feature maps, which are subsequently fed to a CNN for extracting convolutional features in the pixel space. Our neural line rasterization module is designed in a differentiable way for end-to-end learning. We perform experiments on existing large-scale sketch recognition datasets and show that the RNN-Rasterization design brings consistent improvement over CNN baselines and that Sketch-R2CNN substantially outperforms the state-of-the-art methods.

Research Area(s)

  • Freehand sketching, RNN, CNN, neural rasterization, object classification, QuickDraw

Citation Format(s)

Sketch-R2CNN : An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition. / Li, Lei; Zou, Changqing; Zheng, Youyi; Su, Qingkun; Fu, Hongbo; Tai, Chiew-Lan.

In: IEEE Transactions on Visualization and Computer Graphics, Vol. 27, No. 9, 9068451, 01.09.2021, p. 3745-3754.

Research output: Journal Publications and Reviews (RGC: 21, 22, 62)21_Publication in refereed journalpeer-review

Download Statistics

No data available