Dr. Christopher Hopkins, music (PI)
Dr. Kenneth Mark Bryden, mechanical engineering (co-PI)
Karl Svec, Graduate Research Assistant (HCI program)
Jeffrey Kosinski, Undergraduate Research Assistant

Past RAs: Robert Penn Taylor, Andrew Johnson

Support for the initiation of this project was provided by the 2007 Interdisciplinary Collaborative Research Grant from the Center for Excellence in the Arts and Humanities at Iowa State Univeristy of Science and Technology.

Description

VE-SoundSynth maps haptic interaction with objects in a three-dimensional virtual environment to sound synthesis. It is a multi-modal system (vision, sound, touch) that allows a user to control parameters of a sound synthesis engine, and therefore to control variable features of sound, according to contact with the geometries existing a virtual scene. The intended applications of this system are virtual engineering, using sonic feedback to assess engineering design and decision processes, and electroacoustic music, using interaction with virtual scenes as new instruments for musical expression in composition and performance.

The user holds the stylus of a haptic device (Sensible Omni) and moves this in three dimensions, much as a computer mouse is moved in two dimensions. This moves a cursor, or "proxy point" within a virtual scene to navigate the space between objects, for example navigating the scene shown above, which is composed of an array of spheres with a representation of the stylus itself also present in the scene. When the proxy comes into contact with an object in the scene, the motion of the stylus is restricted by force feedback to provide to the user a sensation of having touched that object. Contact with the object then sends messages to the sound synthesis engine that identify the object and the starting coordinates of the proxy upon the surface of the object. These data are then used to set values for the sound synthesis engine, which for example gate an amplitude value to initiate a sound, set waveform parameters such as frequency and phase, and update values used in downstream signal processing. As the user moves the point of contact, 'feeling one's way' upon the surface of the object, messages are sent to update the coordinate values, which in turn modify the generation and modification of sound. Thus the user performs changes of sound by manipulating the position of the proxy on the object much in the manner of using a two-dimensional touch pad, but instead moving upon a virtual 3-D controller the shape of which is determined by the design of the scene for the virtual environment.

The more fully developed scenes used in VE-SoundSynth contain multiple objects in arrays of identical, similar, or dissimilar geometry. In the scene shown above, the spheres are identical, which provides an identical class of motor constraint to the user between contacts with any of the objects. The same motion upon different objects may be mapped to different algorithms in the sound synthesis to provide a means of performing related transformations of sound or musical patterns. The user does not vary the pattern of motion but rather applies the same motion upon a different object. Alternately, other scene designs contain arrays of similar objects of different scale, or dissimilar objects of identical scale. These may be mapped in turn to vary the scale of changes to the sound or shape of sound transformation. Of particular interest are scenes that arrange different regular polyhedra the enclosing spheres of which would be of identical radius, each providing therefore the same range of local motion but a different number of surfaces to map to sound synthesis parameters.

Objects in the scenes are also wrapped with texture gradients the values of which also are sent in messages to the sound synthesizer. For example, a smooth sphere can produce different sound than a gritty sphere. Because the force feedback of the haptic device provides the sensation to the user of the different textures, for the above example slippery versus resistive, there is a different performance characteristic to the contact with otherwise identical objects. The objects may have different surface tension, which further varies the experience of contact with the surface, and effects the performance.

VE-SoundSynth has two implementations: (1) as a standalone system for engineers, using and internal sound synthesis engine with preset mappings designed for interaction with engineering models, and (2) as a controller for musical sound synthesis and music control to any synthesis software accepting OSC messages.

System Diagram