Haptic Feedback in Computer Music Performance

A major concern in computer music performance is the lack of intimacy between one’s haptic senses and the sound being produced. Through the use of computer-controlled force and vibrational feedback, a closer relationship between a performer and sound production may be formed. The development of a haptic interface for computer music will allow for the study of relationships between the haptic senses and cognitive musical processing, eventually leading to the creation of a force feedback based MIDI controller. 1 . Introduction. The relationship between a performer and his instrument is complex and intimate. Traditionally, attempts at understanding instrumental performance gestures have been focused on auditory feedback as the performer adjusts and reacts to the music based on what is heard. However, another major component of this interaction resides in the haptic senses. Essentially, the term “haptics” refers to the senses of touch and motion in the skin and muscles. In acoustic instruments, a direct link exists between a performer’s gestural motion and the sound which is created by the instrument. Any instrumentalist can attest to the importance of being able to feel how an instrument reacts to his gestures. In the case of computer-generated and computer-controlled musical instruments, however, this vital link is often severed. 2 . Issues The difficulty in creating effective performance interfaces is complicated by two major issues. First, since sound signals are created electronically, the physical link between the performer and instrument is not a direct factor in the formation and control of the ultimate sound output. That is, performance gestures must be processed through a stage of algorithmic interpretation before the computer reacts by generating sound. The sound that is eventually produced, then, depends not on the gesture, but on the translation of physical movement into electronic information. The second issue is that this intermediate stage tends to be a one-way transfer of information. Haptic information flows from the performer to the computer, but nothing comes back. This leaves the performer with no haptic feedback to which he may react. Both of these problems need to be solved effectively in order to devise interfaces that more closely approximates the feel and performance of traditional acoustic musical instruments. Another area of interest deals with the cognitive relationship between the haptic senses and one’s processing of musical information. Music is an extremely complex cognitive process so before designing and building a successful interface, one should try first to understand how these two modalities of cognition relate to each other. Much study has been devoted to the relationship between haptics and vision, primarily for the purpose of designing haptic displays for the blind. Work has also been performed in general tactile frequency perception, tactile cognition, the tactile perception of sounds, and even the simultaneous perception of tactile and auditory stimuli. However, little study has been done examining the relationship between haptics and music cognition. The sense of touch is very important to a performing musician, whether the sensations are felt in the fingers, foot, lips, or even internally within the body, and it is important to study exactly why these sensations are so vital, particularly within the context of music. 3 . Haptic and Auditory Stimuli Unfortunately, there is a distinct lack of available information on the relationship between haptics and music cognition. It is useful, however, to look at existing work on haptic perception in non-musical fields, including studies on general tactile frequency perception, tactile cognition, the relationship between haptics and vision, the tactile perception of sounds, and the simultaneous perception of tactile and auditory stimuli. In some of his studies, Gescheider presented simultaneous auditory and tactual stimuli with requirements of increased attentional demands. By using more complex stimuli and requesting the subjects to pay close attention to various events during the trials, he was able to test for performance of longer-term memory and decision-making processes. The results showed that in these situations, the presence of simultaneous auditory and tactual stimuli did not affect perceptual processing, but had strong negative effects on higher level cognitive processing (Gescheider et al., 1975). These results tend to be more interesting when viewed in the context of music due to the higher level processing involved. Perhaps one of the most interesting areas of study involving haptics and sound is a person’s ability to localize sounds based solely on vibrational signals applied to the fingertips. Experiments on this topic involve sending a stereo sound signal to the left and right hands of subjects and having them determine the perceived azimuth of the sound source. These studies have shown that tactile localization improves rapidly with practice and reaches performance levels close to that of hearing. In cases of low frequency sounds, tactile performance may even surpass auditory localization (Gescheider, 1970). Another interesting facet of localization is the ability to project the location of the sound out into the environment (Richardson and Frost, 1979). This suggests that tactile localization makes use of 3-dimensional distance cues in addition to azimuth cues. Vertical localization, however, seems to be very weak in the tactile mode. Also, as in auditory localization, it has been shown that performance in tactual localization improves dramatically when head tracking is included (Frost and Richardson, 1975). In the context of computer music, this could lead to interesting situations in which the performer is provided tactual sound feedback that is closely related to localized auditory sound. 4 . The ProjectThe goal at this stage of the project is to incorporate the above concepts in the design and development of aforce-feedback based MIDI controller. Currently, the apparatus utilizes a 486 PC equipped with an i/o cardfor sensing input from an encoder and sending output to an power amplifier which controls a motor in thecontroller. MIDI information is generated based on input from the encoder and is sent from the PC via aMIDI card to a Macintosh running testing software or music software such as Max. Although thissituation works adequately, the next phase of the project will be to develop a system that needs only aMacintosh in an attempt to simplify the hardware requirements of the controller. When that isaccomplished, it will be possible to use the controller in running studies looking at the relationshipbetween haptic and music perception and cognition. The final stage of this particular project is then to usethe collected information in the creation of a more complex and effective force-feedback controller for use incomputer music performance. 5 . ReferencesCadoz, C. Instrumental gesture and musical composition. ICMC Proceedings 1988, 1-12.Chafe, C. Tactile audio feedback. ICMC Proceedings 1993, 76-79.Frost, B. J. & Richardson, B. L. Tactile localization of sounds: Acuity, tracking moving sources,and selective attention. Journal of the Acoustical Society of America, 1976, Vol. 59 (4), 907-914.Gescheider, G. Some comparisons between touch and hearing. IEEE Transactions on Man-MachineSystems, March 1970, 28-35.Gescheider, G., Sager, L., & Ruffolo, L. Simultaneous auditory and tactile informationprocessing. Perception & Psychophysics, 1975, Vol. 18 (3), 209-216.Gillespie, B. The virtual piano action: Design and implementation. ICMC Proceedings 1994, 167-170.Klatzky, R., & Lederman, S. The Intelligent Hand. Psychology of Learning and Motivation, 1987,Vol. 21, 121-151.Richardson, B. L., & Frost, B. J. Tactile localization of the direction and distance of sounds.Perception & Psychophysics, 1979, Vol. 25 (4), 336-344.Verrillo, R. Vibration sensation in humans. Music Perception, 1992, Vol. 9 (3), 281-302.