Video Game Audio:The who, what, when, where, hows

and whys.

Karen CollinsCanada research chair

collinsk@uwaterloo.ca

1

OutlineOutline

1. WHAT: What is Game Audio?2. WHO: Who cares? Who makes it?3. WHEN: When does the audio team get

involved? WHEN did these processes/techniques develop?

4. WHERE do games differ from linear media?5. HOW and WHY do we fix the problems?

2

WHAT is game audio?WHAT is game audio?

Interface soundsMusicSound EffectsDialogueAmbience Cinematics

all of these can function on several layers (like diegetic/non-diegetic)

3

WHO cares about game WHO cares about game audio?audio?

83% of Adult Gamers listed sound as one of the most important video game console elements

47% of Game Console owners (18-25 yr) hook up their game console to a home theatre system

48% of Hardcore Gamers said surround sound is a purchase driver for next generation consoles

WHO makes it?WHO makes it? The Game Audio Production The Game Audio Production ProcessProcess

Sound DirectorComposer(s)Sound Designer(s)Programmer/Engineer(s)Dialogue/VO director, actors.Licensing/Contracting directorsImplementation specialists

(emerging role)

5

Audio “Post”: IntegrationAudio “Post”: Integration

Sound design, mixing, integration typically handled by the same person.

Director/designer must make decisions regarding implementation of all audio assets (including music)

Often is a programmer as well

6

WHEN does the audio team WHEN does the audio team get involved?get involved?

Sometimes at end of game: populate game with sound

Sometimes at start or in middle: play a larger role in implementation of audio in the game, and can make critical decisions in regards to the development of the game and its sound

7

Music Music drivingdriving gameplay elements.gameplay elements.

8

New Super Mario Bros (Nintendo DS 2006)(Koji Kondo)

WHEN: A Brief History of

Game Audio Innovations

(What’s so different about game audio??)

9

Computer Space!(Nutting Associate

1971)First game to have

sound.

10

Space InvadersSpace Invaders (Midway 1978) (Midway 1978)

First use of continuous “background” music.

11

Technological ConstraintsTechnological Constraints

Up ‘N Down (Sega 1983)

12

Atari VCS (2600)Atari VCS (2600)

Up ‘N Down (Sega 1984)

13

Working With Constraints: Nintendo Working With Constraints: Nintendo NESNES

Metroid (Nintendo 1987) (Hip Tanaka)

vibrato (pitch modulation), tremolo (volume modulation), slides, portamento, echo effects

14

15

BallblazerBallblazer (LucasArts 1984) (Peter Langston) (LucasArts 1984) (Peter Langston)

Algorithmic generation“Riffology” method (Optimized randomness) by

Peter Langston32 eight-note melody fragmentsAlgorithm chooses how fast, how loud, when to

omit notes, when to insert rhythmic break

16

Working With Constraints: Commodore Working With Constraints: Commodore 6464

Track 1 (e.g. “Level_One”)

Instrument 1(envelope, waveform,effect filters, etc.)

Instrument 2(envelope, waveform,effect filters, etc.)

Rob Hubbard’s Module Format

Pattern 1 (sequence of notes)

Pattern 2 (sequence of notes)

Pattern 3 (sequence of notes)

17

Combining modules (in MIDI) with control statements

MIDI and the Creation of iMUSE

Land, Michael Z. and Peter N. McConnell. Method and Apparatus for Dynamically Composing Music and Sound Effect Using a Computer Entertainment System. US Patent No. 5,315,057. 24 May, 1994.

Super Mario WorldSuper Mario World (Nintendo 1991) (Koji Kondo) (Nintendo 1991) (Koji Kondo)

18

Musical layering techniqueMario jumps on Yoshi & gets extra layer of music(SNES).

Legend of Zelda: Ocarina of TimeLegend of Zelda: Ocarina of Time (Nintendo 1999) (Nintendo 1999) (Koji Kondo) (N64)(Koji Kondo) (N64)

Proximity-based algorithms control cross-fades

19

State-of-the-Art TodayState-of-the-Art Today

7.1 to 8.1 surround soundCombination of synth with orchestra, choirAt least 512 channels of sound

God of War (Gerard Marino, Sony 2006)

20

Techniques we can borrow Techniques we can borrow from Filmfrom FilmSpotting a gameEmotional reinforcementUse of sound FX librariesField recording techniques, FoleyMixing of cutscenes/cinematic sequencesUsing multichannel surround*Certain techniques from, for instance,

Ride films—e.g. Prominent sub-woofer, etc.—”physicality” of sound to shock and awe

21

Mix Goals (shared with linear media)Balance: among and between elements—games can’t

undwerstand dialogue through music, explosions, etc.IntelligibilityBelievability: realistic and emotionally effective reduces illusion;

characters for instance suddenly come in much more clearly, or too loud, etc.

  Effective communication: what is the prioirity in the game right

now > focus listener’s attentionPrioritiesGameplay objectives Emotional delivery: audio tells player how to feel.Avoid distractionsAvoid competitions to player’s attention

22

Where games differ from FilmWhere games differ from FilmLinearity Vs. Non-Linearity/ unpredictability!Interactivity with player/multiplayerNo “production” soundTemporality (length)LocalizationBudgetsDelivery methods/technologyListening environmentMixing, dynamic range and the concept of

“post”

What is Interactive Audio?What is Interactive Audio?

sound events occur in direct reaction to a player’s movements. The player triggers the cue, and can repeatedly activate it, such as by making a character jump up and down.

Interactive Audio:Interactive Audio:

FootstepsEnemy music cue

What is What is adaptiveadaptive audio? audio?

“adaptive” audio is generally referred to as sound that reacts to transformations in the gameplay environment—such as a change from day to night set by a game’s timer mechanism. Adaptive audio is not directly triggered by a player

Adaptive AudioAdaptive Audio

Change in music day to nightWolf, crow

IMPACT OF IMPACT OF ADAPTIVE/ ADAPTIVE/ INTERACTIVITY ON INTERACTIVITY ON MUSICMUSIC

28

Problem: transitions from cue to cue◦ No “post” =

unpredictable timings

◦ Abrupt jumps are jarring

Solution: transitions: Cross Solution: transitions: Cross fadefade

Cross FadesCross Fades

31

Other solution: The Other solution: The Stab/SFXStab/SFXSound effect (e.g. Explosion,

gunshots, etc.) or stab (quick shock chord) used to cover transition time

32

Solution: Variable Solution: Variable structurestructure

Create “transition matrix”: chart out all possible directions for each sequence, and create transitions for marker points/jumps

Parameter Based Music: ParametersParameter Based Music: Parameters

Number/action of non-playing charactersNumber/action of playing charactersActionsLocations (place, time of day, etc.)Scripted or unscripted eventsPlayer health or enemy healthDifficultyTimingPlayer properties (skills, endurance)Bonus objects Movement (speed, direction, rhythm)“Camera” angle

34

The transition matrix approach and the creation of transitional units

Example: Parameter-Based MusicExample: Parameter-Based Music

No One Lives Forever (Guy Whitmore 2000)Six standard music states are based on

number of NPC enemies:

1. Silence 2. Super ambient 3. Ambient 4. Suspense/sneak 5. Action/combat 1 6. Action/combat 2

35

Example: Example: No One Lives ForeverNo One Lives Forever

Earth Orbit: Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient: in

elevator) then transitions to

music state 6 (combat 2)

36

Problems: Multi-player interactivityUnscripted events!parameter-based music is difficult:

◦Music cannot be tied to specific events or locations

◦Music cannot be tied to specific parameters

◦To some extent, this is a general audio problem in MMOGs—as audio is as much about emotion as realism.

IMPACT OF IMPACT OF INTERACTIVITY ON INTERACTIVITY ON SOUND EFFECTSSOUND EFFECTS

38

Problems with Game Sound Problems with Game Sound EffectsEffects

Ambience: Loops are boring. Loops are boring. Loops are boring.

Brain picks out patterns.Users get bored with hearing same

sounds BUT sound designers can’t possibly record enough variations of sounds (time, budget)

Users need a new experience every time they play the game (promised by LucasArts’ Euphoria technology)

Audio not responding to physics

Solution: Granular SynthesisSolution: Granular Synthesis

40

Granular Synthesis ExamplesGranular Synthesis Examples

1. Crowd2. Tennis3. Speech

41

Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School

Granular: Remaining Open QuestionsGranular: Remaining Open Questions

What elements in a sound effect can be varied while still maintaining the “meaning” of the sound?

How little of a sound can be changed to alter perception?

How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game?

How to develop an “audio physics engine”: e.g. footsteps change based on how much player is carrying, etc.

42

MIXING IN GAMESMIXING IN GAMES

43

MixingMixingProblems:Problems:-too many sounds-too many soundspoor dynamic rangepoor dynamic range-poor variation-poor variation-unpredictable -unpredictable timingstimings -no “post -no “post production” mixing production” mixing : results in “muddy”, : results in “muddy”, clash of soundsclash of sounds

Dynamic rangeDynamic range

45

… in a popular film

… in a popular game

Graphics adapted from those supplied by Rob Bridgett of Swordfish Studios.

Dynamic Range: ProblemsDynamic Range: ProblemsCompetition with wildly varying levels of

other games: reference levels not effective for games—no widespread adoption of standards

producer/game designer tends to always want it “louder”

Listening environment in games more competition for attention—larger “noise floor” = more competition in acoustic environment- restricts dynamic range

listener fatigue

46

Mixing: Dynamic Range Mixing: Dynamic Range SolutionsSolutions

Reduce amount of soundsVariable VolumeEvangelize need for good dynamic

rangeUse what standards you can find

(e.g. Xbox Boot sound)Be aware of possible noise-makers

◦Non-game sounds: e.g. windows messenger pops up

Variable volumeVariable volume

Solutions: Listener fatigueSolutions: Listener fatigueLearn how to use silence!Noise floors—what is loudest background

noise in environment?Balance impact opportunity with intelligibilityAudio ‘breaths’ (overall/in specific

frequencies—lose some sound e.g. bass to give a rest in particular frequency ranges)

Timeouts or gradual ducking for some elements (if it’s not important, time it out)

VARIABILITY!

Mixing Problem: Dynamic Mixing Problem: Dynamic Range/ VariationRange/ VariationChange DSP rather than just

volume to change audio ◦E.g. reverb to make it softer,

dreamier◦Phasing to create a “dazed” effect◦Overdrive to make it more

aggressive, etc.

50

Solution: Location-Based Run-Time Solution: Location-Based Run-Time MixingMixing

Real-time DSP to adjust sound E.g. bottle drop on hard floor of kitchen or in

next carpeted roomFactor in 5.1 surround to adjust real-time

panning

REQUIREMENT: audio engine to pass parameters from game and from player back and forth to engine.

51

Problem: Mixing: Unpredictability, Problem: Mixing: Unpredictability, VariabilityVariability

52

Problem: Ambience Problem: Ambience repetitionrepetitionNo “production sound” recorded–

ambience assembled from scratch.

Repetition of certain aspects can be detected by brain: patterns easy to discern and takes away from immersion

53

Solutions: Real-time Weighted Solutions: Real-time Weighted MixingMixing

Weighted permutations ◦ Predict which sounds can recur without making

obvious.Example:

◦ Dialogue, Sound FX A. Sound FX B, player sounds, music, ambience

◦ If dialogue = “run!”, set parameter to 1◦ If gunshot is coming towards us, set parameter to 2◦ If no action, fade out music and raise ambience

REQUIREMENT: “intelligent” Engine to predict and set weighting

54

The big problem: LogjamsThe big problem: LogjamsLittle definition for which

elements are acoustically ‘most important

Too much in similar frequency ranges

Intelligibility vs. impact

55

Logjam solutionsLogjam solutionsControl frequencies of soundsAssign priorities in code

◦E.g. dialogue = priority “1” Ambience = priority “4” Gunshot = priority “2” Music = priority “3” Footsteps = priority “5”

Real-time ducking: Use notch filters to cut out frequencies that conflict (must organize in advance!)

56

Steps to organizationSteps to organizationGroup sounds into multi-level categories:

e.g. first-person sounds, third-person, which are first-person weapon, third-person dialogue, etc.

Enforce RMS/peak levels (level limits) per category of sounds—e.g. enemies can never get above -10dB, first person sounds never above -15dB, etc.

Balance categories of sounds amongst themselves—within each category—check to maintain balance level-wise for dialogue, etc.+ choose reference piece

57

Steps to organizationSteps to organizationPrioritize Sound Groups by Function

◦E.g. Goal-relevant sounds, Instructive dialogue, Sounds that impact the player, Enemy weapon fire, Sounds that provide environment or emotional support, Musical score, Ambience, etc.

Dynamic Priorities (Per-Sound)◦E.g. Distance / distance ranking, Total

attenuation, Actual current peak/RMS, Number of playing instances, Game-specific priority schemes

 

58

Steps to organization: the Steps to organization: the MixMixDucking

◦Basic: Notification-driven◦Duck when new higher priority sound

starts◦Duration/speed to ducked state◦Target volume/percentage◦Advanced: Peak/RMS-driven◦Duck/unduck based on current

monitored levels of higher priority category (attack/release)

 59

The mix: culling/mutingThe mix: culling/mutingCulling/Muting

◦Alternative to simple ducking◦Thins out sound field, Brings more

focus to new sounds◦Stop or mute using release

envelopes/fades◦Bandwidth concerns (mute)◦Restart accuracy (stop)

60

The Mix; EQ and FilteringThe Mix; EQ and Filtering◦Per-category critical frequency range(s)◦Can use offline or realtime analysis◦Notch out these frequencies for other

categories (duration for smoothing)‘Blur’ lower priority soundsCompression/Limiting

◦Can combine attenuation with compression/limiting

◦Keeps low-pri sounds from falling off too significantly or obviously

◦Can conflict with ducking desire

61

MIXING: SurroundMIXING: SurroundSurround as game tool: player

can interact/depend onHRTF: where is the player’s

character> Where are they facing, etc.?

Remember the CHARACTER may be changing position, but the PLAYER may not be! How do you adapt to this??

62

A final point on mixingA final point on mixingMIXING IS MORE THAN REALISM.

◦What are the psychological aspects of mixing?

◦The most effective mix is not always the most realistic.

63

Audio MiddlewareAudio Middleware

64

Wwise Wwise prototype the integration of audio in a game

simulation before the game is even finished.Environmental effects can be rendered in real-time,

and occlusion and obstruction effects can be managed within the software, mixing up to four simultaneous environments per object.

Sound prioritization for real-time mixing is also included, as is randomization of various elements for effects such as pitch or volume, to enhance realism.

Real-time game parameter controls can also be set, to adjust sound properties based on parameter value changes in a game

Validate profiles (how expensive is your audio!?)

65

TECHNOLOGICAL TECHNOLOGICAL CONSTRAINTS CONSTRAINTS (MOBILE)(MOBILE)

66

Problem: technological Problem: technological constraintsconstraintsMOBILE AUDIO:

◦Trying to write songs and create sounds in a few kB for constrained technology such as mobile phones

Cheese RacerCheese Racer

Muting and unmuting multiple tracks in a MIDI file to produce various mixes.◦2 drum loop samples,◦three tones: bass, melody and chords

In the code for the first level, ten different combinations of tracks: percussion + bass, percussion + bass + melody, bass + melody + chords, etc.

(cont..)(cont..)

Notice how the mix changes every time the mouse gets a piece of cheese.

Music loop 40 seconds. 10 different combinations: therefore, each level has more than 6 minutes of different music mixes. “Because the mix changes depending on gameplay, the music will never play exactly the same way twice, thereby increasing variation and decreasing "ear fatigue."”

In fact, there are two more sets of tracks playing two more styles of music, using the same tempo and percussion tracks as the first level.

Entire game contains almost 20 minutes of various music mixes, using only 68K of compressed sample and MIDI data.

Solutions: technological Solutions: technological constraints: constraints:

Playing a single sample at different pitches to produce variation.

The "pitch it up, play it down" technique for saving space.

Sampling to save sizeSampling to save sizeThis game contains only a single "trumpet fall"

sound effectmodify the playback sample rate so that each

time you pick up a piece of cheese, the sound is played at a different pitch

randomly vary the pitch during gameplay, so there's no repeating pattern to it

The sound tends to mask the transition from one mix to another, helping to create a more seamless audio experience.

same kind of pitch-shifting effect applied to car horn beeps: (fig 1)