Data-driven Models of Manga with Applications to Composition and Mobile Viewing

Description

During the past few decades, manga (or Japanese comics) has gained massive popularityacross the world. This has resulted in an increasing involvement of the general population inreading manga, as well as increased interest in amateur artists in producing their own mangastrips and imagery. The success of manga can be mainly attributed to the unique storytellingtechniques used by the artist to enhance the sense of reader participation. Among thesetechniques, the composition of foreground subjects and speech balloons is crucial toproviding readers with a smooth reading experience. Effective composition improves thestorytelling and engages the reader by continuously and consistently guiding readers across apage and through the artworks and dialogue in a semantically meaningful order. However,creating high-quality manga compositions typically requires well-trained skills and uniquetalent, making its creation inaccessible to amateur artists.Unlike Western comics, which has a rigid set of guidelines (e.g., rectangular panels, withspeech balloons on the top), manga are created in a more flexible way, without a set ofpredefined guidelines (e.g., speech balloons surround the speakers). However, if thecomposition of elements does not effectively convey the order of the elements, then there willbe difficulties in reading and comprehending the manga page. Manga artists mainly followtheir own intuition and understanding to create effective layouts and compositions, i.e., theyfollow their own implicit set of rules for manga creation, with variations depending on theartist’s own preferences. However, currently, there is no existing computational model thatdescribes how manga artists use the composition of subjects, speech balloons, andbackground artworks to affect and engage the reader’s attention.The goal of this project is to develop data-driven computational models of manga thatprovide insight into how manga readers view and interact with manga artworks, and to applythese models to automatic manga composition and mobile manga viewing. In particular, weaim to develop a model that jointly represents the composition of manga elements (e.g.,subjects, speech balloons, and background), and its affect on viewer attention (as measuredthrough eye movement patterns). We adopt a data-driven approach, where we first collect alarge dataset of reader behavior while reading manga pages using eye-tracking technology,and then use the dataset to train a probabilistic graphical model representing the jointrelationships between manga elements and attention. When using a mobile device to readmanga, the device screen can be seen as the viewport of the reader’s attention (e.g., panningis analogous to shifting attention). We extend the composition-attention model to alsoincorporate the interaction of the reader on the manga page. The augmented model will betrained from a dataset collected from readers using mobile devices to view manga (recordingtheir touch-based interactions, pans and zooms). Finally, we apply our computational modelsto two manga-related applications: 1) automated manga composition for amateur artists tocreate professional-looking manga pages; 2) a mobile manga viewer that allows the user toread manga pages with a fluid and continuous experience via an interactive animation.Furthermore, our composition model is general and we also consider applications to otherfields, e.g. magazine design, personalized advertisement, and evaluation of mobile websitedesign.