The Influence of Sound and Music on Images

by Bärbel Neubauer and William Moritz


If you have a QuickTime plug-in, you can view and listen to the many clips included in this essay. All clips have been produced by Bärbel Neubauer as an exclusive for Animation World Magazine.

Just as many different techniques create moving visual images, so do many different instrumentations create musical sounds. Images (ie. colors, forms, materials, juxtapositions and contrasts, etc.), and sounds (ie. tones, melodies, rhythms, orchestral colors, harmonies and counterpoints) function the same in different media.

Kiss Basic Kiss

The soundtrack has a lot of influence on the image, and on how we perceive the imagery of a film. It might be an animated or live-action film, long or short, documentary or abstract, music video or narrative -- every film has its own special character in its image and sound, which together forge a dramaturgic development, an "animation" that paces the piece.

Sound and music define and interpret the visual image by expressing mood and directing our attention. Sound is emotionally direct, so it is a powerful tool.

Examples to Consider
I have created some examples of the functions and influence of sound and music on visual imagery. For the basic picture, I made a simple animation of a kiss, a line drawing that will remain open to several interpretations influenced by the use of sound. Further versions with more elaborate colors and animation demonstrate other uses of music and sound. The story always remains the same. See the basic drawing "Kiss Basic Kiss."

First I show how the image directs the sound, or, vice versa, how the sound directs the image. In "Basic Shot 1," the visual image makes the couple kiss. In "Shot 2 + Rhythm," the kiss creates the shot and starts the rhythm. These same systems are used in all kinds of film. In music animations, a very skillful use of both is necessary; just making the image and sound synchronize is not interesting.

Basic Shot 1
Shot 2 + Rhythm

Next in "Kiss Basic Kiss," "Cicadas," "Kiss Basic Start" and "Basic Snore," I concentrate on sounds like atmospheric noises and original sounds, including the storytelling aspect of sound.

Kiss Basic Kiss

"Kiss Basic Kiss" and "Cicadas" both have a naturalistic sound. Please note the difference: the line drawing of "Kiss Basic Kiss" is funnier, so that it could be used as a spot by itself. "Cicadas" works out both image and sound naturalistically, in the action and the surroundings -- it tells all of the piece's details, and would demand a longer film and story. "Kiss Basic Start" has no natural sound, but rather another noise that describes what is going on inside the two heroes -- a subjective mode of storytelling. "Basic Snore" adds a third person to the plot, telling the story from an observer's viewpoint. I've adapted the timing and sequence of animation to this end.

Kiss Basic Start
Basic Snore

All sounds, especially atmospheric noises, can be combined with or included in music. Techno uses many sampled natural noises for rhythm.

Making Music
The third section of examples deals with music as a soundtrack. Music is built by using three main levels: rhythm, layering and melody. Both in music and imagery a certain rhythm must lead through the work and give a constant consistent structure to the piece. The rhythm defines the kind of "dance" that the animation, and music and sound, are performing. It is much easier to watch animation films if they are accompanied or directed by a strong rhythm. The rhythm divides time into many pieces, like frames. It directs our attention when we watch the image moving and hear the music floating. It is a pattern that serves our brain for orientation, recognition, and remembering.

Since this is music, it must not only be a technical pattern, but must also express a certain feeling, by the instrumentation, sequence, length, and dynamics of the beats/tones. Many different instruments may play the rhythm, and which instruments are chosen determines the character of the music. Older classical music often had no separate section of rhythm/drums, but rather played rhythm within the melody using strings and wind instruments. This created a spherical or spiritual mood -- quite contrary to African drums, which have a really strong body. The darker and rounder the bass notes, the more reality and physical power the music gets. The higher, brighter and lighter the sounds are, both in rhythm and the music in general, the more spherical and spiritual it becomes.

Basically natural instruments have more presence in tone body than electronic ones. They sound nearer to the spectator/listener and more personal. Electronic voices do have more space; how wide and cosmic they seem varies with panorama and stereo effects. Using these aspects, you can again define space with sound.

If classical music contains a human voice, it gets more body again, even if there are no drums. (See later "Kiss Classic.")

The speed of rhythms in general takes its orientation from the human heartbeat. Music that vitalizes you, but does not upset you, has a speed like your pulse: over 60 beats per minute, but not more than 70 beats. These numbers may be multiplied in layers, varying with the density of rhythm elements in the music.

Shot 2 + Rhythm
Rhythm + Layer

I made a simple song. "Shot 2 + Rhythm" starts the rhythm. Then "Rhythm + Layer" adds a layer. The layer(s) fills the space of the music. In this case, the layering adds quite a bit of space to the heroes' feelings. Layers are multiple swinging voices, like strings, choruses, etc. Electronically, they are sampled sounds from sound modules or self-sampled noises. Sampling works well for layering, as it establishes large rooms/spaces, and fills them easily. Also, as mentioned above, one can attain additional definition to the event arena by using stereo panorama effects. Layers are said to make it easier to listen to music. On the other hand, layering can also soften and subtract some of the music's profile. It depends on which message you want. If you don't have any layering, the solo voice has to be more elaborate. (Later listen to "Kiss Romance": the single voice of the trumpet is stronger than either a layer, or the trumpet with a layer, would be.)


Influencing the Story
In "Rhythm+Layer+Voice+Major," a piano plays the melody/voice. The voice plays/sings the melody of the music. It tells the story of the piece.

Song in Minor

In "Song in Minor," I used the same music, but played the layering and melody in a minor key: the mood gets sadder and darker. The image is changed a little, too.

Kiss Old Hit

In "Kiss Old Hit" the music interprets the line drawing again, but differently from the other versions. This led to the next step: still using the same story, I worked out the images more specifically for different kinds of music, so the image responds to the sound.


Kiss Jazz

"Kiss Jazz." Jazz music refers to universal/natural laws and expressions. It is rather hard to play, because the scales change very quickly. It is usually underlined by a "walking bass" that goes on and on swinging. The bass is quite differentiated, having many more tunings than the basic tones of the melody. Therefore, the bassist must master all musical knowledge of scales. The piano must also continuously swing and swing, like water. The percussive rhythm section -- drums, hi-hats, brushes, agogos, wooden sticks, etc. -- changes rhythms and instruments frequently. Plus, the human voice is also frequently used like an instrument. In this example, I used a simple instrumentation (though the music is not simple), and a simple pattern of brush dots in the image. With this music, the image seems light in character.

Kiss Romance

"Kiss Romance." The timing is soft and slow. The main colors blue and pink. The music contains no layering. A sense of a large space is created by the very personal feeling and the expressiveness of the trumpet.

Kiss Techno
Basic Shot 1

"Kiss Techno." The music dictates a quicker timing, and strong colors like red, black, orange, etc., as well as hard contrasts, like black and white patterns. This example was worked out on a computer, as a digital medium seemed quite relevant. I used a rough soundtrack, with very little melody, to contrast with the other music. The difference between classical music, for example, would not have to be so big. Techno melodies and harmonies are very similar to classical ones, only slightly reduced and set to a much stronger rhythm. Instrumentation of layers are used percussively, as in the "orchestra hit" in "Basic Shot 1." Also, as in Jazz, the whole swinging can be turned around in reverse, gaining more elastic tension. Certain Techno groups do use elements of Jazz in their compositions and I like that a lot.

Kiss Classic

"Kiss Classic." The voice lends a personal feeling to the whole harmonic building and layering of strings. If we heard only a string ensemble, the music, and therefore, the whole film would have less presence.

Kiss Experimental

"Kiss Experimental." The instrumentation of the music is reduced and humorous with the voice of the song experimenting with the upper tones, acting animal-like and rhythmic. The image is also reduced and graphic, with drawings and colors that are applied abstractly, as well as figuratively.

Kiss Disco

"Kiss Disco." This animation could be used for several different kinds of music. In this example, I used a tension-filled but still ambiguous rhythm for demonstration. The image is -- also ambiguously -- narrative, turning to abstract metamorphoses and back to figurative. In both "Kiss Disco" and "Kiss Techno" I changed from figurative to abstract and back. "Kiss Experimental" combines both abstract and figurative in the same image.

Sound, Color, Imagery and Art
When I was working on my first abstract film in the '80s (I had done live-action and narrative animations before that), I spent about half-a-year painting metamorphoses of colors and forms with oil pastels on small sheets of half-translucent paper while listening to music all day long. All of the images I painted came from the subconscious. Sometimes shapes and colors like creatures appeared in the paintings. I had no concrete plan of what should happen. I just went on painting and it developed itself. After six months, the metamorphoses faded out. I don't know what the "director" of this piece was: whether it was the colors themselves, or if the colors were engendered by the various music to which I was listening. When I began the piece, I had only the "A" of the tuning fork in my mind, with no sense of a time or place, nor any orientation from a particular musical instrument. Musicians tell me that this "perfect pitch" is quite rare, and much appreciated. I made a soundtrack based on it, with a sung triad on this "A" and recorded drops of water. When this project was finished and I began to work on others, my perfect pitch "A" disappeared as suddenly as it had come. This experience told me what I had always known without ever trying to prove scientifically: colors and sounds are the same. They follow the same laws. These laws are natural, and can be used for any purpose.

Meanwhile, I read in a book that the ratio between color and sound frequencies is P x 1,000,000,000; the vibration of colors is faster than the vibration of sounds. The "P" stands for the Greek "Pi" which in geometry represents the relation of the radius of a circle to its circumference. Significantly, abstract animation works with circles a lot. Moreover, we talk about tone colors. It might be nice to construct a computer program that would translate sounds into colors and vice versa. (Maybe it already exists?) It would be of no use for making live-action or animation that involved spoken dialogue and the representation of natural things. It would only offer new ways of experimenting with visual music.

An Austrian writer recorded birds and played their calls at different speeds. She found out that the music of many classical composers can be assigned to different kinds of birds. That was in the '70s. Today we can sample all possible kinds of sounds very easily. Sampling allows one to vary the tempo of vibration. One can use a pure sine-wave as a basic vibration. One can build up an orchestra of natural sounds, using stones or tennis balls for percussion instruments. From these basic frequencies of vibrations one can build up numerous variations by using filters (like color filters...) and by changing the tempo and direction of the vibrations. In this way one can alter the "material," the character and effect of the tone. Sounds that have a lot of high-frequency vibrations like Asian instruments, all natural materials and percussion instruments/materials are very suitable for sampling.

Laurie Anderson has been translating time, movement and dance into sounds electronically in her performances for the past 20 years. Meredith Monk deals with the same artistic issues, using mainly the natural voices of humans for experimenting with the upper frequencies of tones. Her first tapes were of music made entirely with her voice. Now she sings like a bird and growls like an angry animal. The mood of her music is quite precisely defined and the listener is encouraged to generate their own images: singing like a bird suggests very high trees where birds feel at home...

Meredith Monk now performs in theaters, together with musicians and accompanied by surrealistic imagery, dreamlike in their manipulation of light and shadow, but still communicating a lot of serious issues, like aspects of social dialogue and communication.

Long before Meredith Monk, abstract animators like Harry Smith and Hy Hirsh painted films for Jazz concerts to accompany the music; to make the music visible, and the paintings move. For abstract animation both natural and electronic music, or a combination of both, are fine. If one takes natural instruments, it doubles the direct effect. For example, in live-action, it would bring a close-up even closer. If one uses electronic sound, it may establish a brilliant contrast or bring some neutral elements into the work.

Bärbel Neubauer is an independent filmmaker based in München, Germany.

William Moritz teaches film and animation history at the California Institute of the Arts.

Note: Readers may contact any Animation World Magazine contributor by sending an e-mail to

Table of Contents
Past Issues

Animation World Magazine
Career Connections | School Database | Student Corner
Animation World Store | Animation Village | Calendar of Events
The AWN Gallery | The AWN Vault | Forums & Chats

About | Help | Home | | Mail | Register

©1999 Animation World Network