Digital Sound Synthesis - Aalto · Oscillators in Subtractive Synthesis • Usually periodic...

1

Digital Sound Synthesis

Vesa Välimäki and Fabián Esqueda

3.3.2017

Sound check

Course Schedule in 2017

3.3.2017© 2003-2017 Vesa Välimäki and Fabián Esqueda2

0. General issues (Vesa) 13.1.20171. History and future of audio DSP (Vesa) 20.1.20172. Digital filters in audio (Vesa) 27.1.20173. Audio filter design (Vesa) 3.2.20174. Analysis of audio signals (Vesa) 10.2.2017No lecture (Evaluation week for Period III) 17.2.20175. Audio effects processing (Fabian) 24.2.20176. Synthesis of audio signals (Fabian) 3.3.20177. 3-D sound (Prof. Ville Pulkki) 10.3.20178. Physics-based sound synthesis (Vesa) 17.3.20179. Sampling rate conversion (Vesa) 24.3.201710. Audio coding (Vesa) 31.3.2017

2

F Introduction to sound synthesisF Wavetable and sampling synthesisF Additive synthesisF Subtractive synthesisF FM synthesisF Classification of digital synthesis methods

F Two demos: Integrated Wavetable and Texture Synthesis

Outline

33.3.2017© 2003-2017 Vesa Välimäki and Fabián Esqueda

4

Promise of Digital Sound Synthesis• Max Mathews (Science, 1963):“Sound from numbers: ...any perceivable sound can be produced...”

– If any sound can be stored as numbers, it must be possible to compute those numbers

• Goal: produce sound from scratch

3.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

3

5

Applications of Sound Synthesis• Making music

– Musical instrument sounds– Singing voice

• Communication with devices– Alarm, warning, and control sounds

• Computer gaming and virtual reality– Music that changes with circumstances– Gunshots, explosions, collisions, …– Environmental sounds, like wind, rain, …– Vehicles, machines, …

• Test signals– Acoustic measurements– Psychoacoustic and cognitive research


6

What sound?


4

7

What sound?


8

What sound?


Source: Coca-Cola’s “Pop and Pour” by Suzanne Ciani

5

9

Problems in Sound Synthesis1) Software explosion: how to compute sound efficiently?

– Natural synthesis requires a complicated system– If simplified too much, sounds artificial– Computers are getting faster – programming remains challenging

2) Control issues: how to play synthetic sound?– Control data must be obtained from musicians– Solved partly by real-time control interfaces and music software

(e.g., MIDI devices, sequencers)


10

ReacTable• For more info and videos, see: http://www.reactable.com/


Source: http://www.indiewire.com/2014/09/watch-bjo rk-is-c oming -to-a -the ater -nea r-yo u-19 1062/

6

Integrated Wavetable Synthesis

Eero & Pekka

19

Additive Synthesis• Also called Fourier synthesis or

sinusoidal modeling– Produce any sound by combining

sine waves

• Example: piano tone

(Figure from Perry Cook’s homepage: http://www.cs.princeton.edu/~prc)


7

20

Additive Synthesis, cont’d• Each partial is generated

separately• Accurate control, but lots of

data!!!– User rarely needs to control

everything that accurately

(Figure from Perry Cook’s homepage: http://www.cs.princeton.edu/~prc)


21

Extensions of Additive Synthesis• Sines + noise modeling (Serra

and Smith, 1990)– Sine waves for the deterministic part– Filtered noise for the stochastic part

• Sines + noise + transients (Verma and Meng, 2000)– To avoid smearing of attacks, model

transients separately

• FFT–1 synthesis (Rodet and Depalle, 1992)– Use the inverse FFT to compute

waveform based on spectral data


8

22

Resynthesis using Sines + Noise

Original

Sinusoids

Residual noise Filtered white noise

Signals available on-line at http://www.cs.tut.fi/sgn/arg/music/tuomasv/sinusoids.html3.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

23

Sines + Transients + Noise (Verma and Meng, 2000)

• Sinusoids are unsuitable for synthesizing transients!

Demo by Paulo Esquef, 2001


9

24

West Coast vs East Coast


25

West Coast vs East Coast Synthesis


Don Buchla (1937–2016) Bob Moog (1934–2005)

10

26

West Coast Synthesis


• Started by Don Buchla in the 1960s• First Buchla synth built for the San

Francisco Tape Music Center• Synthesizers aimed for experimental work,

break from traditional paradigms• No keyboards, sequencer or pressure-

sensitive plates instead• Many cool synth makers (e.g. Make

Noise, Intellijel, Doepfer, etc.) are keeping West Coast synthesis alive!

27

West Coast Synthesis


• Start with a signal waveform that has a poor spectrum (e.g. a sinewave) • Expand its spectrum using nonlinear waveshaper or frequency

modulation• Process signal using Lowpass Gate (LPG)

0 1/f0 2/f0Time (s)

(a)

-1

-0.5

0

0.5

1Vin SPICE Model

0 1/f0 2/f0Time (s)

(b)

-1

-0.5

0

0.5

1

Example: Wavefolder!

Check Arturia’s Microbrute’s feature “Metalizer” for cool examples.

11

28

Buchla’s Legacy


“For the past four decades, Buchla instruments have consistently set the standards for innovative musician interfaces that provide performer-friendly access to an enormous wealth of sonic resources. Don’s new modular synthesizer system, the 200e, continues this tradition. The panel layouts are nothing short of elegant, while the underlying functions are the most advanced and musically rich available today.” – Dr. Robert Moog

Link to the video I showed in class:

➤ Kaitlyn Aurelia Smith Buchla Music Easel. Source: https://www.youtube.com/watch?v=Uhtar8FlgzU

29

Subtractive Synthesis (East Coast)


• Started by Robert Moog in the 1960s• Began with publication of the Voltage-

Controlled-Filter (VCF) at AES• Moog’s aim was to attract musicians to

new sounds• Designs built around keyboard interface• Most Roland and Korg synths also fall

within this category.

Moog System 55 ->

12

31

Sound Generation in Subtractive Synth• Based on source-filter model• Start with a signal waveform that has a rich spectrum• Filter it with a lowpass filter or resonator

• A better term ‘source-filter synthesis’ is used in speech modeling– Nothing is subtracted, really

Filterx(n) y(n)


32

Implementing Subtractive Synthesis• One or more oscillators

– Different fundamentals (e.g., octave apart) or slight detuning for phasing

• Second- or fourth-order resonant lowpass filter• At least two envelope generators (ADSR)

(Sound example by Antti Huovilainen, 2005)3.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

13

33

Oscillators in Subtractive Synthesis• Usually periodic waveforms

– Waveform contains all harmonics or only odd harmonics of a fundamental

• Digital implementation must suppress aliasing

(Figure from: T. D. Rossing: The Science of Sound. Second Edition. Addison-Wesley, 1990.)


34

• Trivial sampling leads to harsh aliasing particularly at high fundamental frequencies– Inharmonicity– Beating– Heterodyning

Aliasing – The Movie

Video by Andreas Franck, 20123.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

14

35

• Additive synthesis of harmonics below the Nyquist limit only

Sawtooth Sweep Without Aliasing

Video by Andreas Franck, 20123.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

36

Aliasing in Sampled Sawtooth Waveform• Trivial sawtooth

– Much aliasing

• Additive synthesis of harmonics(1/f spectrum)– Ideal case– No aliasing

• Difference of the two: the aliased signal


15

37

Digital Oscillator Algorithms1.Bandlimited synthesis methods

• Additive synthesis & its variants (wavetable synthesis, DSF)

2.Quasi-bandlimited synthesis methods• ‘BLIT’ - Bandlimited impulse train and filtering (Stilson & Smith, ICMC’96)• ‘MinBLEP’ - Minimum-phase bandlimited step (Brandt, ICMC’01)• ’PolyBLEP’ – Polynomial bandlimited step (Välimäki & Huovilainen, 2007)

3.Alias-suppressing synthesis methods• Oversampling• Abs[sin(x)] with a shaping filter (Lane et al., Computer Music J., 1997)• ‘DPW’ - Differentiated parabolic wave (Välimäki, IEEE SPL, Mar. 2005)

4.Post-processing methods• Suppress aliasing by filtering (Pekonen & Välimäki, IEEE ICASSP, 2008)


38

Differentiated Parabolic Wave Algorithm• An innovative way to generate a sawtooth wave with reduced

aliasing (Välimäki, IEEE Signal Processing Letters, March 2005)– Only 2 parameters: fundamental frequency f and sampling frequency fs

H(z) = c (1 – z–1)where c = fs/4f


16

39

Signal Generation in DPW Algorithm

0 10 20 30 40 50-1

0

1

0 10 20 30 40 500

0.5

1

0 10 20 30 40 50-1

0

1

Discrete time

• Output of modulo counter x(n)– A ‘trivial’ sawtooth wave

• Squared signal x2(n)– Piecewise parabolic wave

• Differentiated signalc [x2(n) – x2(n–1)]– Difference of neighbors


40

Aliasing is Reduced!

• Spectrum of modulo counter signal x(n)

• Spectrum of squared signal x2(n)

• Spectrum of differentiated signalc [x2(n) – x2(n–1)]

0 5 10 15 20-60

-40

-200

Leve

l (dB

)

0 5 10 15 20-60

-40

-20

0

Leve

l (dB

)

0 5 10 15 20-60

-40

-20

0

Leve

l (dB

)

Frequency (kHz)

Nyquist limit(22050 Hz)Desired spectral components O


17

41

Compare Sawtooth Wave Algorithms• A scale at high fundamental frequencies

– Trivial sawtooth (modulo counter signal)

– DPW sawtooth– Ideal sawtooth (additive synthesis)

fs = 44.1 kHz


43

Rectangular Pulse Generation Using DPW• Two alternative methods

(Välimäki & Huovilainen 2006)

(a) Subtract two sawtooths– Phase shift adjust pulse width– Easy pulse width modulation

(b) Use inverse comb filter to copy the sawtoothwith phase shift, then subtract

– Requires interpolated delay line for smooth pulse width modulation


18

44

Higher-order DPW Oscillators• Trivial sawtooth can be integrated multiple times to improve the

DPW method (Välimäki et al., 2010)

The polynomial signal must be differenced N – 1 times and scaled


45

Integrated Polynomial Waveforms

N = 1

N = 3

N = 5

N = 2

N = 4

N = 6


19

46

Differenced Polynomial Waveforms

N = 2

N = 4

N = 6

N = 1

N = 3

N = 5


47

Differenced Waveforms

N = 2

N = 4

N = 6

N = 1

N = 3

N = 5


20

48

Polynomial Transition Region (PTR)• The PTR algorithm implements DPW efficiently and extends it

Ref. Kleimola and Välimäki, 2012.

q Trivial sawtooth (modulo counter)

• DPW waveform

Constant offset

Sampled polynomialtransition region

3.3.2017© 2003-2017 Vesa Välimäki and Fabián Esqueda

49

Fabian’s favorite subtractive synths!

3.3.2017

© 2003-2017 Vesa Välimäki and Fabián Esqueda

21

50

Virtual Analog Synthesis• Digital real-time simulation of analog synthesis

– Digital waveform generation (avoiding aliasing)– Digital 2nd and 4th-order filters– Good SNR– Programmability– Nostalgia– Musicians like the knobs !

(Source: http://www.clavia.se)

• First product in 1995: Nord Leadby Clavia (Stockholm, Sweden)

– Currently many products by various manufacturers (Yamaha, Korg, Access, Novation, Roland, …)


51

(Source: http://www.clavia.se)3.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

22

57

FM Synthesis• Invented by John Chowning in late 1960s

– Patent and publication in the 1970s (Chowning, 1973)– Products since early 1980s, – Patent was owned by Yamaha until about 1994– Still going strong (synthesizers; previously sound cards, phones)

• FM = Frequency Modulation– The same idea as in FM radio

• Another cheap way to compute intriguing sounds


58

Basic Idea in FM Synthesis• Very fast vibrato is applied to a sine wave (“carrier”)

– Vibrato generator is called “modulator”– Both are now on audio frequencies

• Demo: speeding up the vibrato when fc = 200 Hzfm = 0 Hz, 2 Hz, 10 Hz, 20 Hz, 50 Hz, 100 Hz, 200 Hz

x(t) = A sin[ωct + I sin(ωmt)]

⊕

fc

VCO = Voltage-Controlled Oscillator

VCA = Voltage-Controlled Amplifier

Modulator

Carrier (Fig

ure

from

: T. D

. R

ossi

ng:

The

Sci

ence

of

Sou

nd.

Sec

ond

Edi

tion.

Add

ison

-Wes

ley,

1990

.)


23

59

Generation of Harmonics in FM Synthesis• Sidebands are equidistant in frequency just like harmonics• Spectrum can be predicted using Bessel functions

(Figure from: K. Steiglitz, A Digital Signal Processing Primer with Applications to Digital Audio and Computer Music. Addison-Wesley, 1996)


60

Modulation Index in FM Synthesis• Number of harmonics is proportional to modulation index I

– Here modulation index I is increased from 1 to 15

(Figure from: K. Steiglitz, A Digital Signal Processing Primer with Applications to Digital Audio and Computer Music. Addison-Wesley, 1996)3.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

24

61

Spectrum of FM Synthesis• Frequency components get mirrored, if they occur above the

Nyquist frequency or below 0 Hz• Inharmonic sounds when the frequency ratio between carrier

and modulator is not an integer • For interesting results, time-varying parameters are needed• Good for bell-like and metallic timbres

– Example: fc/fm = 1/1.4 and I decreases exponentially:


62

Aliasing in FM Synthesis• I = 2• f_m = 200 Hz• f_c = 750 … 100 Hz

Video by Juha Karjalainen& Sampsa Lehtonen, TKK, 2006


25

63

Yamaha DX7• The first affordable digital music synthesizer• Only FM synthesis

– Well suited to producing metallic timbres (electric piano, tubular bells etc.)

• A huge commercial success in mid-1980s

(Source: www.obsolete.com/120_years)3.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

64

Modern Classification of Digital Sound Synthesis Methods (Smith, 1991)

1) Abstract algorithms– FM synthesis, subtractive synthesis, Karplus-Strong algorithm

2) Processed recordings – Musique concréte, wavetable synthesis, sampling

3) Spectral modeling– Formant synthesis, sinusoidal modeling (additive synthesis),

sines+noise modeling

4) Physical modeling– Various approaches (to be discussed next week…)


26

65

Taxonomy of Digital Sound Synthesis Methods (J. O. Smith, 1991)


66

• Computational tricks that produce complex signal– Not based on any physical sound-production mechanism

of nature– Not based on properties of hearing

• Typically brilliant and simple algorithms that produce interesting sound in a restricted way– Experts can recognize by hearing which technique is used

• FM synthesis, waveshaping, subtractive synthesis, the Karplus-Strong algorithm– Vector Phaseshaping synthesis (VPS) (Kleimola et al. 2011)

distorts sine waves by playing part of the period faster, the rest slower…

1. Abstract Algorithms


27

67

• Digital recording, modification, and playback– Musique concrète goes digital

• Today, sampling and wavetable synthesis are the most popular synthesis techniques– Memory is cheap!– Large fast memories are available

• Samplers, digital pianos, sound cards, software synthesis taking advantage of a large hard disk, polyphonic ring tones in mobile phones

• Example: Synful, a sampling synthesis software for expressive orchestral music (2005-)– http://www.synful.com/

2. Processing of Recordings


68

• Humans analyze sound in the frequency domain– Spectrum is the main property of musical sound

• Spectral models focus on modeling spectrum of sound– Particularly the frequency and amplitude of harmonic

components, which may vary over time

• Computationally demanding, but almighty DSP– Everything is possible … in theory– Cross-synthesis: borrow spectral envelope from another sound

• Formant synthesis, sinusoidal and sines+noise models and FFT-1 (modern additive synthesis)– For example, the CHANT singing synthesizer (IRCAM, Paris,

1984): http://www.ircam.fr/anasyn/reine.html

3. Spectral Models


28

69

• Imitate the sound production principle of a sound source– For example plucked string, brass instrument, drum, motor etc.

• Motto: Sound will be natural, if the model is right!– However, if the model is too simple, only some features are natural

(happens easily in practice)– Functioning of the model can be very intuitive: easy control

• This is the topic of another lecture

4. Physical Models


70

Recent Singing Voice Synthesis• Vocaloid singing synthesis

software by Yamaha– See http://www.vocaloid.com/

and http://www.zero-g.co.uk/• Personal singing synthesis

– Based on spectral concatenation and sinusoidal modeling

– First Leon and Lola, then Miriam (first released in 2004)

• Score editor – Melody, words, expression

• Great for backing vocals, (almost) natural for lead vocals

Lola Choir of Lolas


29

71

Vocaloid 2 (2007), Vocaloid 3 (2011)• V2 was a big commercial

success– Best selling software in Amazon

Japan for some time– Improved sound quality

• Marketing assisted by a virtual celebrity, Hatsune Miku– Check out her greatest hit on the

YouTube: “Ievan polkka”, an old Finnish folk song (> 14 million views)

Link: http://en.wikipedia.org/wiki/Hatsune_Miku3.3.2017 © 2003-2017 Vesa Välimäki and Fabián Esqueda

72

• There exist many digital synthesis methods– The most popular are sampling and virtual analog– Many instruments can be synthesized well

• Sound modeling synthesizers are coming– The golden era of samplers will end…

• Non-musical sound synthesis will rise– Electric vehicle sounds (danger at slow speeds!)– Simulator sounds (e.g. working machines)– Sound for virtual reality (cf. silent films to talkies) – Procedural audio (algorithms for any sound)

• Computers must learn to produce all sounds– Audio must catch up computer graphics

Conclusions and Future


30

• Select and study one of the scientific articles listed below ØCompact summary (approx. 2-3 pages) and your own thoughtsØ Implement and explore some algorithm presented in the paper using

MatlabØReport your findings

• Deadline: Thu 16.3.2017 at15.45 (in 2 weeks)

ASP-2017 Homework 3

§ H.-M. Lehtonen, V. Välimäki, and T. I. Laakso, “Canceling and selecting partials frommusical tones using fractional-delay filters,” Computer Music Journal, vol. 32, no. 2,pp.43–56, Summer 2008.

§ V. Välimäki, J. S. Abel, and J. O. Smith, “Spectral delay filters,” Journal of the AudioEngineering Society, vol. 57, no. 7/8, pp. 521–531, July/Aug. 2009.

§ M. Fink, M. Holters and U. Zölzer, “Signal-matched power-complementary cross-fadingand dry-wet mixing”, in Proc. 19th Int. Conference on Digital Audio Effects, (DAFx-16),Brno, Czech Republic, September5-9, 2016.

• Max Mathews and Julius O. Smith III “Methods for synthesizing very high Qparametrically well behaved two pole filters” in Proc. Stockholm Musical AcousticConference (SMAC), 2003.

§ A. Degani, M. Dalai, R. Leonradi and P. Migliorati, “Time-frequency analysis of musicalsignals using the phase coherence” in Proc. 16th Int. Conference on Digital Audio Effects,(DAFx-14), Maynooth, Ireland, September 2-5, 2013.

§ F. Eichas and U. Zölzer, “Black-box modeling of distortion circuits with block-orientedmodels”, in Proc. 19th Int. Conference on Digital Audio Effects, (DAFx-16), Brno, CzechRepublic, September 5-9, 2016.

ASP-2017 Homework 3

31

75

References• J. Chowning, “The synthesis of complex audio spectra by means of frequency modulation,” J. Audio

Eng. Soc., vol. 21, no. 7, pp. 526–534, Sept. 1973.• M. Mathews, “The digital computer as a musical instrument,” Science, vol. 142, no. 3592, pp. 533–

557, Nov. 1, 1963.• J. Kleimola, V. Lazzarini, J. Timoney, and V. Välimäki, “Vector phaseshaping synthesis,” in Proc.

Int. Conf. Digital Audio Effects (DAFx-11), pp. 233–240, Paris, France, Sept. 2011.• X. Serra, “Musical sound modeling with sinusoids plus noise,” in C. Roads et al. (eds.), Musical

Signal Processing. Swets & Zeitlinger, 1997. Available on-line: http://www.iua.upf.es/~xserra/articles/msm/

• J. O. Smith, “Viewpoints on the history of digital synthesis,” in Proc. Int. Computer Music Conf.(ICMC’91), pp. 1–10 (keynote talk), Montreal, Canada, Oct. 1991. A revised version is available on-line at:http://www-ccrma.stanford.edu/~jos/kna/

• T. Tolonen, V. Välimäki, and M. Karjalainen, Evaluation of Modern Sound Synthesis Methods. Report no. 48, Lab. of Acoustics and Audio Signal Processing, Helsinki Univ. of Tech., Espoo, 1998. Available at: http://www.acoustics.hut.fi/~ttolonen/sound_synth_report.html

• T. Verma and T. Meng, ”Extending spectral modeling synthesis with transient modeling synthesis,”Computer Music J. vol. 24, no. 2, pp. 47–59, Summer 2000.


76

• A. Huovilainen, “Non-linear Digital Implementation of the Moog Ladder Filter,” in Proc. 7th Int. Conf. Digital Audio Effects (DAFx'04), pp. 61-64, Naples, Italy, October 5-8, 2004. Available online at: http://dafx04.na.infn.it/WebProc/Proc /P_061.pdf

• J. Kleimola and V. Välimäki, “Reducing aliasing from synthetic audio signals using polynomial transition regions,” IEEE Signal Processing Letters, vol. 19, no. 2, pp. 67–70, Feb. 2012.

• Pekonen, J., and Välimäki, V., ''Filter-Based Alias Reduction in Classical Waveform Synthesis,'' in Proc. ICASSP'08, pp. 133-136, Las Vegas, Nevada, March 2008.

• V. Välimäki, “Discrete-Time Synthesis of the Sawtooth Waveform with Reduced Aliasing,” IEEE Signal Processing Letters, vol. 12, no. 3, pp. 214-217, March 2005.

• A. Huovilainen and V. Välimäki, “New Approaches to Digital Subtractive Synthesis,” in Proc. Int. Computer Music Conf. (ICMC’05), Barcelona, Spain, pp. 399-402, Sept. 2005.

• V. Välimäki and A. Huovilainen, “Oscillator and Filter Algorithms for Virtual Analog Synthesis,” Computer Music J., vol. 30, no. 2, pp. 19-31, summer 2006.

• V. Välimäki & A. Huovilainen, “Antialiasing Oscillators in Subtractive Synthesis,” IEEE Signal Processing Magazine, vol. 24, no. 2, pp. 116–125, Mar. 2007.

• V. Välimäki, J. Nam, J. O. Smith, and J. S. Abel, “Alias-suppressed oscillators based on differentiated polynomial waveforms,” IEEE Transactions on Audio, Speech and Language Processing, May 2010.

Virtual Analog Synth Papers by AALTO


Digital Sound Synthesis - Aalto · Oscillators in Subtractive Synthesis • Usually periodic...

Documents

Transcript of Digital Sound Synthesis - Aalto · Oscillators in Subtractive Synthesis • Usually periodic...