In this thesis I discuss my empirical investigation into how meaning-making is realized in the process of reading multimodal visual narrative. Japanese shonen manga (adventurous comic story for teenagers), a particular genre of visual narrative that is well-known in the Pan-Asian area has been selected as the target of research because of its popularity and accessibility.

Halliday’s Systemic Functional Theory (SFT) has been adopted for this study because of (a) its metafunctional orientation which covers both microscopic and macroscopic aspects of manga as multimodal discourse, (b) its greater tolerance for multisemiotic expressions, and (c) the importance it places on the interactive context in which the communication takes place (see Halliday 2002a; 2002b; Halliday & Hasan 1985/1976).Manga, whose components cover a multimodal cline from graphics to verbals, is seen as a multimodal narrative whose features are projected onto the different metafunctions: time — experiential, with diegetic time lapse defining the start point to construe experience; cause-effect — logical, with relative positioning of components reflecting logical relations diegetically; space — compositional, with diegetic and extradiegetic spaces contributing to coherence and cohesion of the story (see Bordwell & Thompson 2008 for features of narrative).1 The ideational aspect of SFL focuses attention on the experiential and logical sub-components. In terms of manga, the experiential metafunction is realized by different modes such as word, onomatopoeia, and image; the logical metafunction is embodied through positioning and grouping various modes together.

More verbal elements such as words and onomatopoeic expressions display stronger connection to time elapse in the story for they represent either flowing conversation or 1 Diegetic: within the story. See Chapter II for details. other on spot physical action and phenomenon, and are therefore marked as positive for Diegetic Time Lapse (+DTL).2 The more graphic elements such as images, on the other hand, do not incur direct perception of time and are thus negative for time lapse (-DTL). The manga components labeled as +DTL carry primarily narrative information, whereas those marked -DTL express mainly descriptive information.

The textual/Compositional metafunction enables analysis of comics in terms of those components contributing to coherence and cohesion. For example, by changing the shape of panel frame or the color of gutter, the creator may establish the contrast between different tenses, or between objective and subjective points of view. Compositional aspects of comics depict a consistent and continuous world in the story, which has an overarching time line and a corresponding space.

The interpersonal aspect of manga makes it a medium of exchange between the creator and the reader and encourages the reader to actively decode idiosyncratic meanings. The interpersonal aspect is perhaps the most pervasive metafunction in visual narrative. For manga, the interpersonal metafunction includes such interfaces as drawing style, framing, resemblance, interactive design, etc., and it is crucial to comics study given comics’ commercial nature and stylistic aesthetics.

In this thesis I have attempted to map out the principles for interpreting the reading of visual narrative. In order to acquire first-hand data on actual reading process in a reliable and as unobtrusive manner as possible, eye-tracking sessions together with accompanying interviews have been carried out to collect data on visual narrative reading. SFT provides the top-down theoretical framework, while the bottom-up empirical investigation relies on data extracted from eye-tracking case studies using Japanese manga as stimuli. Through examining and analysing eye movements of the respondents, I have observed that fixations and saccades function on multiple metafunctional levels, distinguishing between different modes and weaving them 2 DTL stands for diegetic time lapse, and the +/- mark indicates the presence/absence of DTL. together. Furthermore, features such as diegetic time lapse in the SFT-based framework have been reflected and reinforced by the case studies.

In order to describe and quantify the process of recognizing visual contrasts as modes, I propose a possible approach by defining the Function of Mode. Different concoctions of visual input will be defined as different functional variables that work together to confine an output that is meaningful in visual narrative. Also, the discussion of spatio-temporal realization in manga will be directed to converge with metaphor study so as to provide a larger picture of the diegetic world which forms the context for understanding meaning-making in multimodal visual narrative.