Optimizing Audio for Mobile...
Transcript of Optimizing Audio for Mobile...
![Page 1: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/1.jpg)
Optimizing Audio for Mobile DevelopmentBen HougeBerklee College of MusicMusic Technology InnovationValencia, Spain
![Page 2: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/2.jpg)
Optimizing Audio• Different from movies or CD’s• Data size vs. CPU usage• For every generation, we must optimize• Maximize quality within constraints
![Page 3: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/3.jpg)
Leisure Suit Larry: Love for Sail!• MS-DOS 5.2 or Windows 3.1 or greater• 486 DX2 processor (66 MHz)• 8 MB RAM (DOS), 12MB (Windows)• 22 MB Hard drive space• SVGA, 256 colors• 1996 specs!
![Page 4: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/4.jpg)
Compare with iPhone 6• 1 GB of RAM• 64 bit, dual core 1.38 GHz processor • 16-128 GB of hard drive space• 2014 specs!
![Page 5: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/5.jpg)
Optimizing Audio• Efficient Spotting and Editing for Games• Digital Audio Theory and Formats• Creative Reuse of Audio • File Management and Middleware
![Page 6: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/6.jpg)
Efficient Spotting and Editing for Games
![Page 7: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/7.jpg)
The best way to save space on audio in your game:
Don’t put audio in your game!
![Page 8: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/8.jpg)
Spotting a Game• Look at the information the game knows• Think about what’s important to the user• Don’t assume you need audio all the time• Prototype and iterate• Log your sounds and analyze• Get an audio professional involved early
![Page 9: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/9.jpg)
Editing Sound for a Game• Trim sounds tightly• Edit to zero crossings• Record loud, turn down in software
![Page 10: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/10.jpg)
Audio Quality and File Size
![Page 11: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/11.jpg)
Three Parameters Affect Bitrate• Sample rate• Bit depth• Number of channels
![Page 12: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/12.jpg)
Sample Rate• Sampling an analog signal from a mic• Measurement of amplitude• Linear PCM (uncompressed)• Sampling rate: how often we sample• The Nyquist frequency• CD quality is 44100 Hz
![Page 13: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/13.jpg)
Bit Depth• Resolution of each sample measurement• Lower bit depths increase noise• CD quality is 16 bits
![Page 14: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/14.jpg)
Sample Rate/Bit Depth Demo
![Page 15: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/15.jpg)
Perceptual Audio Coding• Analyze a signal, throw away frequencies we won’t miss.• Lossy compression: some data is gone forever• Varies based on input: noise compresses very little, silence compresses a lot
![Page 16: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/16.jpg)
Examples• MP3• Ogg Vorbis• AAC• WMA/XMA• AC3
![Page 17: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/17.jpg)
Perceptual Audio Coding• Sometimes this can be used for aesthetic purposes• Performance hit for decompressing sounds• Hardware acceleration available on some platforms (iPhone, Xbox One…)
![Page 18: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/18.jpg)
Lossless Audio Compression• Examples
• FLAC• Apple Lossless
• All data is retained• Not ideally suited for games
![Page 19: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/19.jpg)
Compress with Care• Don’t use the same settings for all sounds• Takes more time, but improves quality• Keep hi-res version, work from copy
![Page 20: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/20.jpg)
Creative Reuse of Sounds
![Page 21: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/21.jpg)
Benefits of Reuse• Create new sounds from existing data• Minimize footprint, maximize variation• Map audio parameters on to game data• Close synchronization and immersion
![Page 22: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/22.jpg)
Example: footstep sounds• Choosing from a set of sounds randomly• Real-time manipulation of pitch, volume• Emulating the physics of the natural world• Very efficient in terms of CPU
![Page 23: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/23.jpg)
Example: Combinatoriality• Multiple layers of random sounds• Exponential increase in combinations• Consistency also increases• Still very efficient
![Page 24: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/24.jpg)
Example: Randomizing Loops• Infinite variation from a short loop• Scalability and parameterization• A kind of granular synthesis
![Page 25: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/25.jpg)
Example: Generative Music• Multiple independent layers• Intermittent phrases• Synchronized to a metronome• Cued from game events• Scalable to gameplay
![Page 26: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/26.jpg)
Example: EndWar Loading Music• Extremely limited resources• Little memory, no disk access• Music responded to load times• xxx kb
![Page 27: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/27.jpg)
Real-Time Effects• Sophisticated digital signal processing• Avoids the need for multiple versions• Scalable with game parameters• Infinite variety
![Page 28: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/28.jpg)
Effects in EndWar• Filter and distortion for radio effect• Filter for distance simulation• Audio particle system for explosions
![Page 29: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/29.jpg)
Audio Effects in iOS7kAudioUnitSubType_BandPassFilterkAudioUnitSubType_DynamicsProcessorkAudioUnitSubType_DelaykAudioUnitSubType_AUFilterkAudioUnitSubType_GraphicEQkAudioUnitSubType_HighPassFilterkAudioUnitSubType_HighShelfFilterkAudioUnitSubType_PeakLimiter
kAudioUnitSubType_LowPassFilterkAudioUnitSubType_LowShelfFilterkAudioUnitSubType_MultiBandCompressorkAudioUnitSubType_MatrixReverbkAudioUnitSubType_NetSendkAudioUnitSubType_ParametricEQkAudioUnitSubType_SampleDelaykAudioUnitSubType_Pitch
Even easier to use with new AVAudioEngine in iOS8!
![Page 30: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/30.jpg)
Synthesis• Huge topic with much potential• Like “soft synths” in music production• More CPU usage, but far less data• Opportunities for real-time manipulation• A golden age of MIDI?• Also for sound effects
![Page 31: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/31.jpg)
File Management and Middleware
![Page 32: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/32.jpg)
Data management for game audio• Requires organization• Tracking many small files• Enforce a naming convention
![Page 33: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/33.jpg)
Audio Engine vs. Game Engine• Game engine calls sound events• Audio engine manages sound events• Audio designer defines audio behavior• Clear and efficient division of labor
![Page 34: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/34.jpg)
Possible Audio Behaviors• Play sound, Play multiple sounds, Play sound with variations, Stop sound, Play one sound while stopping another, Change volume on sound, Apply effect, etc.• Audio implementer doesn’t need to know about this; just call the event
![Page 35: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/35.jpg)
Use Middleware• Most common solutions
• Wwise (Audio Kinetic)• FMOD
• Big gains for all but the simplest games• Reasonably priced (in some cases free)• Available for Android, iOS, and others
![Page 36: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/36.jpg)
Middleware Advantages• Mix in-game for a tight iteration loop• Log and profile your audio data use• Avoid redundancies• Export for multiple platforms• Prioritize your sounds• Incorporate effects plug-ins
![Page 37: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/37.jpg)
Conclusion
![Page 38: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/38.jpg)
Summary• Spot your game carefully for audio• Edit your sounds tightly• Compress sounds individually• Reuse your sounds creatively• Track your assets carefully• Mix and profile your sounds iteratively
![Page 39: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/39.jpg)
Questions?Ben HougeBerklee College of MusicMusic Technology Innovation
[email protected]/微博: @AleaBoy
![Page 40: Optimizing Audio for Mobile Developmenttwvideo01.ubm-us.net/o1/vault/gdcchina14/presentations/... · 2015-03-05 · Optimizing Audio for Mobile Development Ben Houge Berklee College](https://reader030.fdocuments.net/reader030/viewer/2022040909/5e80c9c9de3a826b5761295b/html5/thumbnails/40.jpg)
• Click to edit Master text styles• Second level
• Third level• Fourth level
• Fifth level