Large Audio AI Models for Fixed-Media Electronics in "Prelude: To Listening": The use of AI and traditional Chinese instruments in sonic art

This paper explores the innovative integration of Large Audio AI Models (LMs) with traditional Chinese instruments in the concert version of the work "Prelude: To Listening". Through the use of AI-generated audio accompaniment for the instruments sanxian, sanshin, and Chinese percussions, the project pushes the boundaries of sonic art and mixed music. The study examines the creative potential of LMs, such as AudioLDM2 and Audiocraft , in generating novel sounds that deviate from exact replication, thereby offering fresh avenues for artistic expression. The paper highlights the conceptual innovation of using film subtitles as musical prompts, which imbues the audio with additional narrative and emotional layers. It also discusses the challenges of debugging LMs and the limitations of controlling audio features through text, emphasizing the need for more sophisticated methods to bridge text inputs and audio outputs. The performance, led by artist Ryo IKESHIRO and supported by a team of collaborators, showcases the potential of LMs in the evolution of experimental music and sound arts. The code and audio are available at https://github.com/prelude-to-listening/concert.