Adaptive Sinusoidal Modelingspcc.csd.uoc.gr/_docs/Lectures2016/SPCC16_StylianouI.pdf · Adaptive...

Adaptive Sinusoidal Modeling

Yannis Stylianou

University of Crete, Computer Science Dept., Greece,

[email protected]

SPCC 2016

Outline Sinusoidal Modeling Advanced Modeling QHM aQHM eaQHM Applications Key papers References

1 Sinusoidal Modeling

2 Advanced Modeling

3 QHMValidation

4 aQHMDescriptionValidationAM-FM decomposition

5 eaQHM

6 ApplicationsSpeech technologies

7 Key papers

8 References

[email protected] Adaptive Sinusoidal Modeling 2/48

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

Special case, Harmonic Model:

x(t) =K∑

k=−Kγke

j2πkf0t

Estimation of parameters (Linear approach):

γ = Ffk s

Model Evaluation, Mean-Squared Error (MSE):

∫w|s(t)− x(t)|2 dt

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

x(t) =K∑

k=−Kγke

j2πkf0t

γ = Ffk s

∫w|s(t)− x(t)|2 dt

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

x(t) =K∑

k=−Kγke

j2πkf0t

γ = Ffk s

∫w|s(t)− x(t)|2 dt

Sinusoidal Models

Sinusoidal Model:

x(t) =K∑

k=−Kγke

j2πfk t

x(t) =K∑

k=−Kγke

j2πkf0t

γ = Ffk s

∫w|s(t)− x(t)|2 dt

iQHM, aQHM, eaQHM, aHM

Towards Adaptive Sinusoidal Models

Estimating Sinusoidal parameters

Sinusoidal representation for a speech/signal frame:

x(t) =

k=−Kake

j2πfk t

Methods:

FFT-based methods (i.e., QIFFT [Abe et al., 2004-05, [1] [2]])Subspace methodsLeast Squares (LS) method

Frequency mismatch:

fk = fk + ηk

x(t) =

k=−Kake

j2πfk t

Methods:

Frequency mismatch:

fk = fk + ηk

x(t) =

k=−Kake

j2πfk t

Methods:

Frequency mismatch:

fk = fk + ηk

x(t) =

k=−Kake

j2πfk t

Methods:

Frequency mismatch:

fk = fk + ηk

x(t) =

k=−Kake

j2πfk t

Methods:

Frequency mismatch:

fk = fk + ηk

x(t) =

k=−Kake

j2πfk t

Methods:

Frequency mismatch:

fk = fk + ηk

Quasi-Harmonic Model, QHM [5]

Frequency mismatch:

x(t) =

k=−Kake

j2πfk t

QHM (de Prony 1795, Laroche [3] (1989), Stylianou 1993,Pantazis [4] (2008, 2011) ):

x(t) =

k=−K(ak + tbk)e j2πfk t

Quasi-Harmonic Model, QHM [5]

Frequency mismatch:

x(t) =

k=−Kake

j2πfk t

QHM (de Prony 1795, Laroche [3] (1989), Stylianou 1993,Pantazis [4] (2008, 2011) ):

x(t) =

k=−K(ak + tbk)e j2πfk t

A few details of QHM

QHM in the frequency domain:

Xk(f ) = akW (f − fk) + jbk2π

W ′(f − fk)

Decomposition of bk : bk = ρ1,kak + ρ2,k jakThen

Xk(f ) = ak

[W (f − fk)−

2πW ′(f − fk) + j

2πW ′(f − fk)

]and taking into account:

W (f − fk −ρ2,k

2π) = W (f − fk)−

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Approximation of the kth component of QHM

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

][email protected] Adaptive Sinusoidal Modeling 7/48

W ′(f − fk)

Xk(f ) = ak

[W (f − fk)−

2πW ′(f − fk) + j

2πW ′(f − fk)

2π) = W (f − fk)−

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

W ′(f − fk)

Xk(f ) = ak

[W (f − fk)−

2πW ′(f − fk) + j

2πW ′(f − fk)

2π) = W (f − fk)−

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

W ′(f − fk)

Xk(f ) = ak

[W (f − fk)−

2πW ′(f − fk) + j

2πW ′(f − fk)

2π) = W (f − fk)−

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

W ′(f − fk)

Xk(f ) = ak

[W (f − fk)−

2πW ′(f − fk) + j

2πW ′(f − fk)

2π) = W (f − fk)−

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

Approximations in QHM

we found previously:

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

]Back to the time-domain:

xk(t) ≈ ak

[e j(2πfk+ρ2,k )t + ρ1,kte

j2πfk t]w(t)

Initially, we assumed:

xk(t) = ak

[e j(2π(fk+ηk ))t

in other words, it is suggested:

ηk = ρ2,k/2π

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

xk(t) ≈ ak

j2πfk t]w(t)

xk(t) = ak

[e j(2π(fk+ηk ))t

ηk = ρ2,k/2π

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

xk(t) ≈ ak

j2πfk t]w(t)

xk(t) = ak

[e j(2π(fk+ηk ))t

ηk = ρ2,k/2π

Xk(f ) ≈ ak

[W (f − fk −

2π) + j

2πW ′(f − fk)

xk(t) ≈ ak

j2πfk t]w(t)

xk(t) = ak

[e j(2π(fk+ηk ))t

ηk = ρ2,k/2π

Constraints

2π) = W (f − fk)−

2πW ′(f − fk)+

O(ρ22,kW

′′(f − fk))

Influence of the Window

We would like small value for W ′′(f ) at fk

W ′′(f ) is influenced by the length, T , of the analysis window,i.e., for rectangular window: W ′′(f ) ∝ T 3

So ... we would like a short analysis window

But ... long analysis window provides robustness

Length of the window � bandwidth

Influence of frequency mismatch

Assumey(t) = αe j(2πf t+ηt)

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

LS solution provides:

a = αsin(ηT )

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

Frequency mismatch estimate:

η = 3

ηT 2− cot(ηT )

)[email protected] Adaptive Sinusoidal Modeling 11/48

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

a = αsin(ηT )

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

η = 3

ηT 2− cot(ηT )

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

a = αsin(ηT )

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

η = 3

ηT 2− cot(ηT )

modeled by QHM as

x(t) = (a + tb)e j(2πf t) − T ≤ t ≤ T

a = αsin(ηT )

b = α 3j(sin(ηT )

η2T 3− cos(ηT )

ηT 2)

η = 3

ηT 2− cot(ηT )

Combining Influences

−400 −300 −200 −100 0 100 200 300 400−400

−200

η1 (Hz)

−400 −300 −200 −100 0 100 200 300 400−400

−200

η1 (Hz)

RectWinHamming

no iter2 iter

• Iteratively, the bias can be removed when |η| < B/3, where B isthe bandwidth of the squared analysis [email protected] Adaptive Sinusoidal Modeling 12/48

Robustness against additive noise

Signal contaminated by noise:

y(t) =4∑

akej2πfk + v(t)

Mean Squared Error (MSE):

MSE{fk} =1

M∑i=1

|fk(i)− fk |2

MSE{ak} =1

M∑i=1

|ak(i)− ak |2

Comparison with Cramer-Rao Bounds (CRB) and QIFFT(Abe et al. 2004)

10000 Monte Carlo simulations

MSE of frequencies as a function of SNR.

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

MSE of amplitudes as a function of SNR.

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

0 10 20 30 40 50 60 70 8010

10−5

SNR (dB)

CRBno iter3 iterFFT

Notes on QHM

The approximation made in QHM is valid provided that thefrequency mismatch lies in a specific interval.

This interval is a function of the bandwidth of the analysiswindow.

Robustness of QHM against noise was tested and verified.

It can be shown that iterative QHM is equivalent to (anapproximate) Gauss-Newton method.

There are also relations between QHM and ReassignedSpectrum

Notes on QHM

QHM and Gauss-Newton method

Assuming we have N samples of x(t) = ce jωtw(t)

Approximate Gauss-Newton solution for frequency

ω(1) = ω(0) −R

{∑Nt=−N tw2(t)x(t)e−jω

c(0)∑N

t=−N t2w2(t)

x(t) = (c + tb)e jωtw(t)

withω(1) = ω(0) + ρ2

= ω(0) −R

{jb(0)

}where

b(0) =∑N

t=−N tw2(t)x(t)e−jω(0)t∑Nt=−N t2w2(t)

ω(1) = ω(0) −R

c(0)∑N

t=−N t2w2(t)

withω(1) = ω(0) + ρ2

= ω(0) −R

{jb(0)

}where

b(0) =∑N

ω(1) = ω(0) −R

c(0)∑N

t=−N t2w2(t)

withω(1) = ω(0) + ρ2

= ω(0) −R

{jb(0)

}where

b(0) =∑N

Evaluation: RMSE for frequency

B SNR = 0dB, N = 100 (left), N = 500 (right).CRB: Cramer-Rao lower bound for frequency estimation

2 4 6 8 10

10−4

10−3

10−2

Iterations(a)

GNiQHMCRB

2 4 6 8 10

10−4

10−3

10−2

Iterations(b)

GNiQHMCRB

Evaluation: RMSE for amplitude

B SNR = 0dB, N = 100 (left), N = 500 (right).CRB: Cramer-Rao lower bound for amplitude estimation

2 4 6 8 10

10−1

Iterations(a)

2 4 6 8 10

10−1

Iterations(b)

Reassigned Spectrum

Time relocation:

τ = −∂φ(τ, ω)

∂ω= τ −R

(XTw (τ, ω)

X (τ, ω)

)where

XTw (τ, ω) = −N∑

t=−Ntw(t)x(t + τ)e−jω(t+τ)

Frequency relocation:

ω = ω +∂φ(τ, ω)

∂τ= ω + I

(XDw (τ, ω)

X (τ, ω)

)where

XDw (τ, ω) = −N∑

t=−N

dtx(t + τ)e−jω(t+τ)

Reassigned Spectrum and QHM

QHM: x(t) = (a + tb)e jωtw(t) with b = ρ1a + ρ2ja1

Then, it can be shown:

τ = τ +W2

and (in case of using Gaussian windows):

ω = ω +W2

Wk =N∑

t=−Ntkw(t)

Reassigned Spectrum and QHM

QHM: x(t) = (a + tb)e jωtw(t) with b = ρ1a + ρ2ja1

Then, it can be shown:

τ = τ +W2

and (in case of using Gaussian windows):

ω = ω +W2

Wk =N∑

t=−Ntkw(t)

Frequency mismatch estimation error

−600 −400 −200 0 200 400 600−2000

−1000

η1 (Hz)

No Iterations

−600 −400 −200 0 200 400 600−2000

−1000

η1 (Hz)

RS−QHM

From QHM to Adaptive QHM, aQHM [6]

QHM (stationarity assumption):

x(t) =

k=−K(ak + tbk)e2πjfk t

Adaptive QHM (aQHM):

x(t) =

k=−K(ak + tbk)e jφk (t)

φk(t) = 2π

0fk(s)ds + ϕ, t ∈ [0,T ]

is the estimated instantaneous phase.

From QHM to Adaptive QHM, aQHM [6]

QHM (stationarity assumption):

x(t) =

Adaptive QHM (aQHM):

x(t) =

φk(t) = 2π

0fk(s)ds + ϕ, t ∈ [0,T ]

is the estimated instantaneous phase.

From QHM to aQHM; Graphically

tl−T

}ρ2,k

Frame Rate in aQHM

One sample: no interpolation between estimations

Higher rates (i.e., 5ms, 10ms): Interpolation betweenestimates is required:

Amplitudes are linearly interpolatedFrequencies are interpolated with splinesPhases are interpolated by integration of instantaneousfrequency

Example of estimation in aQHM: no iteration(QHM)

200 300 400 500 600 700 800 900 1000 1100900

Time (sample)

instantaneous frequency

estimated

Example of estimation in aQHM: one iteration

200 300 400 500 600 700 800 900 1000 1100900

Time (sample)

estimated

Example of estimation in aQHM: twoiterations

200 300 400 500 600 700 800 900 1000 1100900

Time (sample)

estimated

Validation

Let us consider as example:

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

and frequency mismatch: |fk − ζk | = 35Hz , using Hammingwindow

Consider simplified approaches: Sinusoidal-based analysis(SM)

Mean Absolute Error (MAE) [frame rate: one sample]

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

Validation

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

Validation

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

Validation

x(t) = (11− 340t + 4000t2)e j2π(280t+19500t2)

AM FM (Hz)

QHM, 10ms 0.46 2.15

aQHM (1) 0.008 0.006

SM, 10ms 0.44 3.08

QHM reminder

x(t) =

Instantaneous parameters:

Ak(t) =√

(aRk + tbRk )2 + (aIk + tbIk)2

Φk(t) = 2πfkt + atanaIk+tbIkaRk +tbRk

Fk(t) = fk +1

aRk bIk − aIkb

M2k (t)

= fk +1

2πρ2,k

QHM reminder

x(t) =

Instantaneous parameters:

Ak(t) =√

(aRk + tbRk )2 + (aIk + tbIk)2

Φk(t) = 2πfkt + atanaIk+tbIkaRk +tbRk

Fk(t) = fk +1

aRk bIk − aIkb

M2k (t)

= fk +1

2πρ2,k

QHM and AM-FM decomposition

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

Taylor series expansion of the instantaneous phase of kthcomponent:

φk(t) = 2πζkt +∞∑i=0

φk,it i

Instantaneous frequency of the kth component at t = 0:

νk(0) = ζk +φk,12π

... and previously we had:

Fk(0) = fk +ρ2,k

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

φk,it i

νk(0) = ζk +φk,12π

Fk(0) = fk +ρ2,k

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

φk,it i

νk(0) = ζk +φk,12π

Fk(0) = fk +ρ2,k

AM-FM signal

y(t) =

K(t)∑k=1

ak(t)cos(φk(t)),

φk,it i

νk(0) = ζk +φk,12π

Fk(0) = fk +ρ2,k

Reconstruction errors with QHM, aQHM, SM

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.5

Time (s) (a)

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.1

−0.05

Time (s) (b)

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.1

−0.05

Time (s) (c)

0.05 0.1 0.15 0.2 0.25 0.3 0.35−0.1

−0.05

Time (s) (d)

Original

QHM Recon. Error

aQHM Recon. Error

SM Recon. Error

aQHM: AM-FM decomposition of VoicedSpeech

Signal Reconstruction

x(t) =K∑

Ak(t) cos (Φk(t))

Test on 5 minutes of 20 female and male voiced speech(TIMIT)

Average Signal-to-Error Reconstruction Ratio (in dB)

Male Female

QHM 23.9 29.1

aQHM (3) 29.1 34.1

SM 17.2 21.1

x(t) =K∑

Ak(t) cos (Φk(t))

Male Female

QHM 23.9 29.1

aQHM (3) 29.1 34.1

SM 17.2 21.1

x(t) =K∑

Ak(t) cos (Φk(t))

Male Female

QHM 23.9 29.1

aQHM (3) 29.1 34.1

SM 17.2 21.1

Extended aQHM

Recall aQHM:

x(t) =K∑

Extended aQHM:

x(t) =K∑

k=−K(ak + tbk)α(t)e jφk (t)

AM-FM modeling: aQHM

0 0.05 0.11

Time (s) (a)

0 0.05 0.1600

Time (s) (b)

0 0.05 0.11

Time (s) (c)

0 0.05 0.1900

Time (s) (d)

TrueEstimated

AM-FM modeling: extended aQHM

0 0.05 0.11

Time (s) (a)

0 0.05 0.1600

Time (s) (b)

0 0.05 0.11

Time (s) (c)

0 0.05 0.1900

Time (s) (d)

TrueEstimated

Comparing Adaptive Models

0 0.1 0.2 0.3 0.4−0.5

Time (s)

0 0.1 0.2 0.3 0.4−0.5

Time (s)

0 0.1 0.2 0.3 0.4−0.5

Time (s)

0 0.1 0.2 0.3 0.4−0.01

Time (s)

0 0.1 0.2 0.3 0.4−0.01

Time (s)

Stop modeling

0.05 0.1 0.15−0.05

Time (s)

Original signal /t/

0.05 0.1 0.15−0.05

Time (s)

adaptive QHM reconstruction

0.05 0.1 0.15−0.05

Time (s)

extended adaptive QHM reconstruction

0.05 0.1 0.15−0.05

Time (s)

Sinusoidal reconstruction

Stop modeling

Large scale evaluation

Global Signal to Reconstruction Error Ratio (dB)

Model /p/ /t/ /k/ /b/ /d/ /g/

SM 12.7 12.8 12.4 16.6 14.9 15.3

aQHM 19.9 20.6 21.7 28.3 26.9 27.5

eaQHM 25.4 25.7 27.2 32.9 32.2 32.9

Speech Modifications

High quality pitch scale modification

Within glottal cycle formant tracking and modificationstowards voice conversion

Free pre-echo effect in time scaling

Emotion detection and control

Speech Synthesis

Unit selection based: fast synthesis, smooth trajectories,high quality prosodic modifications, voice conversion

Statistical modeling: naturally smooth trajectories,articulators tracking, adaptation and controllability,modulations

Sinusoidal models work nicely with DNNs

Speech Synthesis

Speech Recognition

Robust to noise filter bank estimation

Data driven dynamic information

High-resolution modulations

Fits well to the DNN framework.

Speech Recognition

Speech Pathology

Jitter and shimmer estimators

Vocal fatigue

Robust estimation in spasmodic dysphonia

Vocal tremor.

Speech Pathology

Vocal fatigue

Vocal tremor.

Speech Pathology

Vocal fatigue

Vocal tremor.

Speech Pathology

Vocal fatigue

Vocal tremor.

Key papers to read

R. McAulay and T. Quatieri: Speech analysis/Synthesis based on a sinusoidal representation IEEE TASSP,Vol.34, No.4, Aug 1986, pp 744-754.

Y. Pantazis, O. Rosec, and Y. Stylianou: Adaptive AM-FM Signal Decomposition with Application toSpeech Analysis IEEE TASLP, Vol.19, No.2, Feb 2011, pp 290-300.

Y. Pantazis, O. Rosec, and Y. Stylianou: Iterative Estimation of Sinusoidal Signal Parameters. IEEE SPL,Vol.17, No.5, May 2010, pp. 461-464

P. Babu and P. Stoica: Comments on Iterative Estimation of Sinusoidal Signal Parameters. Comments onIEEE SPL, pp.1022-1023, Vol.17, No.12, Dec 2010

Y. Pantazis, O. Rosec, and Y. Stylianou: Reply to ”Comments on ’Iterative Estimation of Sinusoidal SignalParameters’ ” IEEE SPL, pp. 1024-1026, Vol.17, No.12, Dec 2010

G. Degottex and Y. Stylianou: Analysis and Synthesis of Speech using an Adaptive Full-band HarmonicModel IEEE TASLP, Vol.21, Issue 10, pp 2085-2095, 2013

M. Abe and J. S. III, “CQIFFT: Correcting Bias in a Sinusoidal Parameter Estimator based on Quadratic

Interpolation of FFT Magnitude Peaks,” Tech. Rep. STAN-M-117, Stanford University, California, Oct 2004.

M. Abe and J. S. III, “AM/FM Estimation for Time-varying Sinusoidal Modeling,” in Proc. IEEE Int. Conf.

Acoust., Speech, Signal Processing, (Philadelphia), pp. III 201–204, 2005.

J. Laroche, “A new analysis/synthesis system of musical signals using Prony’s method. Application to

heavily damped percussive sounds.,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Processing, (Glasgow,UK), pp. 2053–2056, May 1989.

Y. Pantazis, O. Rosec, and Y. Stylianou, “On the Properties of a Time-Varying Quasi-Harmonic Model of

Speech,” (Brisbane), Sep 2008.

Y. Pantazis, O. Rosec, and Y. Stylianou, “Iterative Estimation of Sinusoidal Signal Parameters,” vol. 17,

no. 5, pp. 461–464, 2010.

Y. Pantazis, O. Rosec, and Y. Stylianou, “Adaptive AMFM signal decomposition with application to speech

analysis,” IEEE Trans. on Audio, Speech, and Language Processing, vol. 19, pp. 290–300, 2011.

T. F. Quatieri, Discrete-Time Speech Signal Processing.

Engewood Cliffs, NJ: Prentice Hall, 2002.

Acknowledgments

My colleagues: Yannis Pantazis, Olivier Rosec (Voxygen),Vassilis Tsiaras, Gilles Degottex

My ex-PhD Student: Giorgos Kafentzis

Tom Quatieri and Prentice Hall for gave me the permission touse figures from Tom’s book[7]

THANK YOU

for your attention on this part as well!

Adaptive Sinusoidal Modelingspcc.csd.uoc.gr/_docs/Lectures2016/SPCC16_StylianouI.pdf · Adaptive...

Documents

Transcript of Adaptive Sinusoidal Modelingspcc.csd.uoc.gr/_docs/Lectures2016/SPCC16_StylianouI.pdf · Adaptive...