An innovative interface for generating 3D paintings and sound through body motions


Student thesis: Master's Thesis

View graph of relations


  • Chi Chung TANG

Related Research Unit(s)


Awarding Institution
Award date15 Jul 2004


This research project does not just focus in computer science, but also in art. This computer-art project creates a new human-computer interface that captures human body motion gestures and then transforms them into 3D paintings and sounds in real-time. In order to facilitate the seamless interaction between human and the machine, a new computer-vision-based motion tracking systems called Frontal Infrared Illuminated Motion Tracking and Analysing System is introduced. This system can track human body motion in the 3D space in real-time. The users are not required to wear any special clothing, marker or device on their bodies. No constraint is imposed to the users. Two cameras with infrared filters are used such that the 3D information can be extracted. With the use of infrared, the silhouette of the human body can be extracted efficiently and precisely. Based on a carefully designed mapping between the human body motion and visual representation, we can translate the acquired human motion data into virtual 3D colour-path-brushstrokes in a virtual 3D canvas. All of these are done in real-time. The users can produce colourful brushstrokes as they wish by moving their bodies. This is a new media tool for drawing by using body as a brush. This is the Body-Brush. Besides, another mapping is designed for the relationship between body motion and sound. With this mapping, human body motions are transformed into sound. User can generate sound with their whole body intuitively. People can use their body as a baton to control the sound. And this is the Body-Baton.

    Research areas

  • Computer simulation, User interfaces (Computer systems), Interactive computer systems, Three-dimensional imaging, Computer graphics, Human-computer interaction, Human locomotion