3 Sound Synthesis Processing
-
Upload
emmanuel-traveler -
Category
Documents
-
view
31 -
download
3
description
Transcript of 3 Sound Synthesis Processing
-
Sound Synthesis and Processing
Xavier Serra
Music Technology Group
Universitat Pompeu Fabra, Barcelona
-
Xavier Serra - 2008
Index
Tradition of sound synthesis
Musique Concrte, Granular synthesis, Sampling
Spectral sound processing
Mosaicing, Concatenative synthesis
Gesture-based synthesis
-
Xavier Serra - 2008
Tradition of sound synthesis
-
Xavier Serra - 2008
4
-
Xavier Serra - 2008
-
Xavier Serra - 2008
graphical interface for Tbone (Perry Cook)
Synthesis with physical models
-
Xavier Serra - 2008
Digital implementation of an acoustic system
-
Xavier Serra - 2008
Musique concrte
Pierre SchaefferPierre Schaeffer, Etude aux chemins de fer (1948)
A class of electronic art music produced from editing together fragments of natural and industrial sounds.
-
Xavier Serra - 2008
Granular synthesis
All sound is an integration of grains, of elementary
sonic particles, of sonic
quanta Xenakis (1971).
Helmuths example
-
Xavier Serra - 2008
Sampling
Fairlight (1980)Pioneered two innovations that transformed music making, namely sampling and sequencing.
-
Xavier Serra - 2008
Spectral processing
Transform. Spectral
Synthesis Fourier
Analysis Original Sound
TransformedSound
Original Spectrum
TransformedSpectrum
Feature Extraction
Feature Addition
Original Feature
Transformed Feature
Spectral Analysis
-
12
( ) ( )=
++=N
kkkk tfAAtx
10 2cos pi
=
++=N
k
tfjktfjk kk eX
eX
X1
2*
20 22
pipi
wherewhere kjkk eAX
=
( ){ }
+=
+=
=
=
+
N
k
tfjjk
N
k
tfjk
kk
kk
eeAA
eAA
1
20
1
20
Re
Re
pi
pi
Spectral representation
-
13
( ) ( ) ( ) ( ) ( ){ },,,,,,,,,0, 2*22122211*12111210 fXfXfXfXX A spectrum is a sequence of pairs of numbers:A spectrum is a sequence of pairs of numbers:
( ) ( ) ( )2500cos83200cos1410 pipipipi +++= tttxThe spectrum of:
-
14
Spectral analysis:Spectral analysis:
-
15
Short-Time Fourier Transform (STFT)Short-Time Fourier Transform (STFT) Sinusoidal ModelSinusoidal Model Sinusoidal plus Residual ModelSinusoidal plus Residual Model Higher Level AttributesHigher Level Attributes
Spectral models
-
16
soundsound
additiveadditivesynthesissynthesis
sinusoidalsinusoidalcomponentcomponent
**windowwindowgenerationgeneration
smoothingsmoothingwindowwindow
FFTFFTphasephasespectrumspectrum
magnitudemagnitudespectrumspectrum
spectralspectralapproximationapproximation
residualresidualspectral dataspectral data
FFTFFT
magnitudemagnitudespectrumspectrum
phasephasespectrumspectrum
peakpeakdetectiondetection peakpeak
datadata
pitchpitchdetectiondetection
peakpeakdatadata
F0F0frequencyfrequency
amplitudeamplitudecorrectioncorrection
peakpeakcontinuationcontinuation
sine frequenciessine frequenciessine magnitudessine magnitudessine phasessine phases
**
windowwindowgenerationgeneration
smoothingsmoothingwindowwindow
--residualresidualcomponentcomponent
Spectral analysis
-
17
sine datasine data
spectral dataspectral data
amplitude amplitude andand
frequencyfrequencydetrendingdetrending
attack,attack,steady-state,steady-state,and decayand decaydetectiondetection
inflexioninflexionpointspoints
spectral shapespectral shapeextractionextraction
spectralspectralshapeshape
amplitude normalizationamplitude normalizationandand
amplitude envelopeamplitude envelopeextractionextraction
amplitude andamplitude andamplitude amplitude envelopeenvelope
frequency normalizationfrequency normalizationandand
pitch envelopepitch envelopeextractionextraction
pitch andpitch andpitch envelopepitch envelope
loop pointsloop pointsselectionselection
residualresidualsine datasine data
residualresidualspectral dataspectral data
loop pointsloop points
vibratovibratoextractionextraction
vibratovibratodatadata
Feature analysis
-
18
IFFTIFFT++complexcomplexspectrumspectrum
sinesinefrequenciesfrequencies
sinesinemagnitudesmagnitudes
sinesinephasesphases
spectral dataspectral data
spectral spectral sinesine
generationgeneration
magnitudemagnitudespectrumspectrum
phasephasespectrumspectrum
polar to polar to rectangularrectangularconversionconversion
spectralspectralresidualresidual
generationgeneration
magnitudemagnitudespectrumspectrum
phasephasespectrumspectrum
**
windowwindowgenerationgeneration
soundsound
synthesissynthesiswindowwindow
**
spectral dataspectral data(ex: reverberation)(ex: reverberation)
polar to polar to rectangularrectangularconversionconversion
Spectral synthesis
-
Xavier Serra - 2008
Spectral transformations
Original Sound
Transformation
Amp
f fi-1 fi fi+1 fi+2
desired timbre envelope
-
Xavier Serra - 2008
Sound collage
John Oswald
A phonograph in the hands of a hip hop/scratch artist produces sounds which are unique and not reproduced - the record player becomes a musical instrument. A sampler is simultaneously a documenting device and a creative device, in effect reducing a distinction manifested by copyright.
no official distributor/not for sale 1989
-
Xavier Serra - 2008
Photo mosaicing
Example of photomosaic made out of hundreds of portraits of Americans who have died at war in Iraq during the last few years.
-
Xavier Serra - 2008
Audio mosaicing
(Schwarz, 2007)
Tristan Jehan, 2005
-
Xavier Serra - 2008
Concatenative synthesis (1 of 2)
( ) ( )11
1 2 22 1 1
n n
n n
i ii i n nn n
f transp f transpt
i t pitch transp
pi
pi
+ =
+ +
2 c
syncn n
tpitch transp
pi
=
PHASE TIMBRE
stretchedspectrum
1-SSIntp
SPP Spectrum n-i
differentialenvelope
Spectral Shape
Interpolation Factor
(SSIntp)
1Right EpRLeft EpR
Stretch Stretch
Spectral ShapeInterpolation Factor
Frame n-k Frame n-1
Interpolation zonek frames
Frame n
f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping
Timbre Envelope
stretchedspectrum
1-SSIntp
SPP Spectrum n-i
differentialenvelope
Spectral Shape
Interpolation Factor
(SSIntp)
1Right EpRLeft EpR
Stretch Stretch
Spectral ShapeInterpolation Factor
Frame n-k Frame n-1
Interpolation zonek frames
Frame n
f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping f
f
F0n-1 F1 n-1 F2 n-1 F3 n-1
F0 n-1
F1 n-1
F2 n-1
F3 n-1
EpR frequency mapping
Timbre Envelope
-
Xavier Serra - 2008
Concatenative synthesis (2 of 2)
Apossible sounds produced by the instrumentBsounds produced by the performer playing the instrument
recorded audio samples
Instrument sonic space
performance trajectory generator
performance DB
sound rendering
Performance trajectory
performer model
Performance Score
Sound
Synthesizer diagram
Vocaloid example
-
Xavier Serra - 2008
Synthesis based on gestures
gestures generatedby the system
-
Violin gesture modeling