Dissonance and Roughness in Cross-Modal Perception

In music cognition, dissonance has been linked to musical features such as functional harmony and external contexts (e.g. working memory). However, researchers have also suggested that low-level psychoacoustic features, such as auditory roughness, are important to dissonance perception. Similarly, cross-modal mapping has been shown to be unconscious, and we posit that roughness and consequently dissonance perception may be involved in cross-modal processing. We conducted two studies examining the effects of perceived dissonance and computed roughness on cross-modal percepts. In Study 1, we explored possible linkages between visual and auditory constructs. Participants (N = 20) rated eight auditory stimuli for dissonance, perceived shape (curvy to spiky), texture (rough to smooth) and hardness (hard to soft) on a continuous scale. Each stimulus consisted of a stringed or flute instrument playing a single note, at a standard pitch. A linear mixed-effects regression showed significant effects of perceptual dissonance on texture (p = .002) and shape (p = .03). We employed MacCullum’s (2006) computational model to estimate auditory roughness, but no significant effects were found, possibly due to the narrow range of roughness values in instrumental sounds. Thus, in Study 2, another set of eight stimuli were generated, with four predetermined levels of roughness: none (0), low (0.1), medium (0.5) and high (1.0), across two fundamental pitches (110Hz and 220Hz). Roughness was manipulated through the adjustment of modulation index and harmonicity in frequency-modulation synthesis. Participants (N = 41) rated each stimulus for dissonance and undertook a two-alternative forced choice (2AFC) task, matching an image from a pair of graphics depicting a spiky object (previously rated as rough and hard) and a curvy object (smooth and soft) to the auditory stimulus. Findings from a logistic mixed-effects regression suggests that auditory roughness has a stronger effect on the graphic choice, with higher roughness linked to the spikier graphics (p < .001). No significant effects were found for dissonance. In Study 1 participants rated each audio stimulus along a verbal descriptive dimension, and with the exception of round-spiky, most of these items were conventionally used metaphors in describing timbre. However, in Study 2, participants performed a 2AFC between two visual objects different on these dimensions, and tested a direct audio-visual correspondence being sound and sight. As the two tests showed different patterns of effects, future investigations will need to clarify what is the discrete contribution of sensation and language to the cross modal experience of sound.