Harmonic plus Noise models

Outline HNM Analysis Synthesis Further Work Thanks References

Yannis Stylianou

University of Crete, Dept of Computer [email protected]

IIT Guwahati, Dec 2016

[email protected] Harmonic plus Noise, HNM 1/59

1 HNMIntroduction to HNMs

2 AnalysisFrequencyMaximum Voice FrequencyPhase based estimation of pitchAmplitudes and PhasesResidual

3 Synthesis4 Further Work

Error FunctionLeast SquaresTotal Least SquaresNon Linear Least Squares, NLSEnergy modulation function

5 Thanks6 [email protected] Harmonic plus Noise, HNM 2/59

Background

Mentioning just a few works for speech ...

MBE (Griffin et al.1988 [1])

Sinusoids plus band-pass random signals (Abrantes etal.1991 [2]

Harmonic and Stochastic Model (Laroche et al.1993 [3]

Iterative decomposition of the excitation signal(Yegnayarayana et al.1995 [4])

Harmonic plus Noise Model (Stylianou et al.1995 [5]

Harmonic plus Noise Model 2 (Stylianou, PhD, 1996 [6])

Background

Why to decompose?

Decomposing speech into (quasi)periodic and non-periodic parthas many applications in:

Speech modification

Speech coding

Pathologic voice detection (i.e., HNR ...)

Psychoacoustic research

Why to decompose?

Speech modification

Speech coding

Why to decompose?

Speech modification

Speech coding

Why to decompose?

Speech modification

Speech coding

Motivation for HNM

0 200 400 600−2

(a) Time in samples

deOriginal speech signal

0 2000 4000 6000 8000−50

(b) Frequency (Hz)

Original magnitude spectrum

0 200 400 600−2

(c) Time in samples

Harmonic part (0−5000Hz)

0 200 400 600−2000

−1000

(d) Time in samples

Noise part (5000−8000Hz)

Brief overview of HNM

HNM is a pitch-synchronous harmonic plus noiserepresentation of the speech signal.

Speech spectrum is divided into a low and a high banddelimited by the so-called maximum voiced frequency.

The low band of the spectrum (below the maximum voicedfrequency) is represented solely by harmonically related sinewaves.

The upper band is modeled as a noise component modulatedby a time-domain amplitude envelope.

HNM allows high-quality copy synthesis and prosodicmodifications.

HNM in equations

Harmonic part:

h(t) =

L(t)∑k=−L(t)

Ak(t)e j kω0(t) t

Noise part:n(t) = e(t) [v(τ, t) ? b(t)]

Speech:s(t) = h(t) + n(t)

Models for periodic part

HNM1: Sum of exponential functions without slope

h1[n] =

L(nia)∑k=−L(nia)

ak(nia)e j2πkf0(nia)(n−nia)

HNM2: Sum of exponential function with complex slope

h2[n] = <

L(nia)∑k=1

Ak(n) expj2πkf0(nia)(n−nia)

Ak(n) = ak(nia) + (n − nia)bk(nia)

with ak(nia), bk(nia) to be complex numbers (amplitude andslope respectively). < denotes taking the real part.

Models for periodic part

HNM1: Sum of exponential functions without slope

h1[n] =

HNM2: Sum of exponential function with complex slope

h2[n] = <

L(nia)∑k=1

Ak(n) expj2πkf0(nia)(n−nia)

Ak(n) = ak(nia) + (n − nia)bk(nia)

with ak(nia), bk(nia) to be complex numbers (amplitude andslope respectively). < denotes taking the real part.

Models for periodic part continuing

HNM3: Sum of sinusoids with time-varying real amplitudes

h3[n] =

L(nia)∑k=0

ak(n)cos(ϕk(n))

ak(n) = ck0 + ck1 (n − nia)1 + · · ·+ ckp (n − nia)p(n)

ϕk(n) = εk + 2π kζ (n − nia)

where p(n) is the order of the amplitude polynomial, which is, ingeneral, a time-varying parameter.

Residual (Noise) part

The non-periodic part is just the residual signal obtained bysubtracting the periodic-part (harmonic part) from the originalspeech signal in the time-domain

r [n] = s[n]− h[n]

where h[n] is either h1[n], h2[n], or h3[n].

Initial fundamental frequency

Get an initial estimation of fundamental frequency f0 [7]

Determine the voicing of the frame:

∫ 4.3f0

(|S(f )| − |S(f )|)2

∫ 4.3f0

|S(f )|2

Initial fundamental frequency

Get an initial estimation of fundamental frequency f0 [7]

Determine the voicing of the frame:

∫ 4.3f0

(|S(f )| − |S(f )|)2

∫ 4.3f0

|S(f )|2

Maximum Voiced Frequency

Let’s assume that for a voiced frame, a frequency fd has alreadybeen declared as voiced or unvoiced, then

In [fd + f0/2, fd + 3f0/2] find the fc frequency with thehighest magnitude A.For fc compute the cumulative amplitude Ac

Determine all other peaks fi in [fc − f0/2, fc + f0/2], and thecorresponding amplitudes, A(fi ) and cumulative amplitudesAc(fi )Compute the average cumulative amplitude for all fi : Ac(fi )Pass fc through the voicing testSet fd equal to fc and repeat the steps until fs/2.Determine voiced and unvoiced spectral areasMaximum voiced frequency is the maximum frequency of thefirst voiced spectral area.

Voicing Test

Voicing Test:

Ac(fi )> 2

or|A−max {A(fi )}| > 13db

if fc is really close to the closest harmonic lf0, then

declare fc as voiced frequency. Otherwise, declare fc asunvoiced frequency.

Maximum voiced frequency example

0 1000 2000 3000 4000 5000 6000 7000 80000

(a) Frequency (Hz)

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000−2

(b) Number of samples

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

(c) Number of samples

Fundamental frequency refinement

Using the initial f0 value and the L detected voiced frequencies fi ,then the refined fundamental frequency, f0 is defined as the valuethat minimizes the error:

E (f0) =L∑

|fi − i · f0|2

Refinement frequency example

0 500 1000 1500 2000 2500 3000 3500 400020

(a) Frequency in Hz

0 500 1000 1500 2000 2500 3000 3500 400020

(b) Frequency in Hz

Another way to find the initial pitch [8]

Ifx(n) ≈ A e j(ω n+θ+u(n))

withφ(n) = ω n + θ + u(n)

, then by differentiating phase

∆(n) = φ(n + 1)− φ(n) = ω + u(n + 1)− u(n)

we may get an estimation of ω.

Testing estimation

−10 −5 0 5 10 15 20 25 300

SNR (dB)

Applying it to speech ...

HilbertBand Pass filts(n) Transform

ωEst.

Arg{x*(n) x(n+1)}

w(n) =32N

N2 − 1

[n − (N/2− 1)

Results with noisy speech

Tests with noisy speech [8]

0 0.5 1 1.5 2 2.50

Time(sec)

0 0.5 1 1.5 2 2.50

Clean Speech

SNR 10 dB

SNR 0 dB

Amplitudes and phases estimation

Having f0 estimated for voiced frames, amplitudes and phases areestimated by minimizing the criterion:

nia+N∑n=nia−N

w2[n](s[n]− h[n])2

where nia = ni−1a + P(ni−1

a ), and P(ni−1a ) denotes the pitch period

at ni−1a .

for HNM1 and HNM2, this criterion has a quadratic form andis solved by inverting an over-determined system of linearequations.

For HNM3, however,a non-linear system of equations has tobe solved.

Amplitudes and phases estimation

Having f0 estimated for voiced frames, amplitudes and phases areestimated by minimizing the criterion:

nia+N∑n=nia−N

w2[n](s[n]− h[n])2

where nia = ni−1a + P(ni−1

a ), and P(ni−1a ) denotes the pitch period

at ni−1a .

for HNM1 and HNM2, this criterion has a quadratic form andis solved by inverting an over-determined system of linearequations.

For HNM3, however,a non-linear system of equations has tobe solved.

Harmonic Amplitudes and Phases: HNM1

We recall that:

h1[n] =

or in matrices [9]h1 = P1x1

where:

[b−L

... b−L+1... b−L+2

... · · ·... bL

[e j2πkf0(t ia−N) e j2πkf0(t ia−N+1) · · · e j2πkf0(t ia+N)

]Tx1 =

[a∗−L a∗−L+1 a∗−L+2 · · · aL

][email protected] Harmonic plus Noise, HNM 22/59

We recall that:

h1[n] =

or in matrices [9]h1 = P1x1

where:

[b−L

... b−L+1... b−L+2

... · · ·... bL

[e j2πkf0(t ia−N) e j2πkf0(t ia−N+1) · · · e j2πkf0(t ia+N)

]Tx1 =

[a∗−L a∗−L+1 a∗−L+2 · · · aL

][email protected] Harmonic plus Noise, HNM 22/59

We recall (with n′

= (n − nia)):

h2[n] = <

L(nia)∑k=1

(ak(nia) + n

′bk(nia)

)expj2πkf0(nia)n

or in matrices [9]

h2 = P2x2

P2 = [B1|B2]

(B1)nk = E (n−N)(k)

(B2)nk = (n − N) (B1)nkx2 = [<{a0} a1...aL<{b0} b1...bL]

and E = exp(j2πf0(nia))[email protected] Harmonic plus Noise, HNM 23/59

We recall (with n′

= (n − nia)):

h2[n] = <

L(nia)∑k=1

(ak(nia) + n

′bk(nia)

)expj2πkf0(nia)n

or in matrices [9]

h2 = P2x2

P2 = [B1|B2]

(B1)nk = E (n−N)(k)

(B2)nk = (n − N) (B1)nkx2 = [<{a0} a1...aL<{b0} b1...bL]

and E = exp(j2πf0(nia))[email protected] Harmonic plus Noise, HNM 23/59

Solutions for HNM1 and HNM2

x1 =(P1

hWTWP1

)−1P1

Matrix to invert is Toeplitz.

x2 =(P2

hWTWP2

)−1P2

Matrix to invert is block Toeplitz.

where s denotes the vector of the original speech samples

s = [s[−N] s[−N + 1] · · · s[N]]T

and W denotes a diagonal matrix of weights

w = [w [−N]w [−N + 1] · · · w [N]]T

x1 =(P1

hWTWP1

)−1P1

x2 =(P2

hWTWP2

)−1P2

s = [s[−N] s[−N + 1] · · · s[N]]T

w = [w [−N]w [−N + 1] · · · w [N]]T

x1 =(P1

hWTWP1

)−1P1

x2 =(P2

hWTWP2

)−1P2

s = [s[−N] s[−N + 1] · · · s[N]]T

w = [w [−N]w [−N + 1] · · · w [N]]T

x1 =(P1

hWTWP1

)−1P1

x2 =(P2

hWTWP2

)−1P2

s = [s[−N] s[−N + 1] · · · s[N]]T

w = [w [−N]w [−N + 1] · · · w [N]]T

An iterative approach is adopted:

Step 1: Determine phase parameters using HNM1

Step 2: Determine amplitude parameters using least squaressimilar to that for HNM1 and HNM2

back to Step 1

Usually 1-2 iterations are enough.

Avoiding ill-conditioning

For HNM1 there is no problem if window length is twice thelocal pitch period

Same thing for HNM2

For HNM3 stands the same in case the maximum voicedfrequency is less than 3/4 of the sampling frequency and orderof amplitude polynomial is 2

Same thing for HNM2

Residual Signal

The residual signal r [n] is estimated by

r [n] = s[n]− h[n]

Time domain characteristics of r [n]

0 100 200−1

−0.5

1.5x 10

(a) Number of samples

Original speech signal

0 100 200−3000

−2000

−1000

(a) Number of samples

Error signal using HNM1

0 100 200−3000

−2000

−1000

(c) Number of samples

Error signal using the HNM2

0 100 200−3000

−2000

−1000

(d) Number of samples

Error signal using HNM3

Spectral domain characteristics of r [n]

0 1000 2000 3000 4000 5000 6000 7000 8000−20

Residual from HNM2

0 1000 2000 3000 4000 5000 6000 7000 8000−20

Frequency in Hz

Residual from HNM3

... and after adding noise

0 1000 2000 3000 4000 5000 6000 7000 8000−20

(a) Frequency in Hz

dB(a) Residual from HNM2

0 1000 2000 3000 4000 5000 6000 7000 8000−20

(a) Frequency in Hz

(a) Residual from HNM3

Modeling error

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−2

Time in sec

(a) Original signal

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1−60

Time in sec

Variance of the residual signal

The variance of the residual signal is given as:

E (rrh) = I−WP(PhWhWP)−1PhWh

0 10 20 30 40 50 60 70 80 900

Time in samples

Variance using the three harmonic models

Modeling the residual signal

Full bandwidth representation using a low-order (10th) ARfilter

Time-domain characteristics of the residual signal are modeledusing deterministic functions

For all HNMs

nis ←→ nia

For the periodic part: Overlap-and-Add

For the stochastic (noise) part):

Instead of AR coefficients we use reflection coefficientsSample-by-sample filtering of Gaussian noise using normalizedlattice filteringModulation in time with a deterministic function (i.e.,triangular)

For all HNMs

nis ←→ nia

For all HNMs

nis ←→ nia

For all HNMs

nis ←→ nia

For all HNMs

nis ←→ nia

For all HNMs

nis ←→ nia

For HNM1 specifically

for Periodic part (as an alternative to OLA)

Direct frequency matching

Linear amplitude interpolation

Linear phase interpolation using average pitch value

Reformulate the error function

Cost function:

ε(a−L, ..., aL, f0) =1

N∑n=−N

(e[n])2 =1

wheree[n] = w [n](s[n]− h[n])

[e[−N] e[−N + 1] ... e[N]

Reformulate the error function

ε(a) =1

2(s− Ea)hW2(s− Ea)

wherea =

[a−L ... a0 ... aL

e j2π(−L)f0(−N)/fs ... e j2πLf0(−N)/fs

e j2π(−L)f0(−N+1)/fs ... e j2πLf0(−N+1)/fs

......

e j2π(−L)f0N/fs ... e j2πLf0N/fs

(2L+1×2N+1)

Least Squares

Setting:

∂ε(a)

∂a= 0⇒ EhW2Ea− EhW2s = 0

Solution:aLS = (EhW2E)−1EhW2s

Properties:

Asymptotically efficient even when the noise is colored.Rather fast, O(L(N + L)).Assumes no errors in E matrix.

Least Squares

Setting:

∂ε(a)

Properties:

Least Squares

Setting:

∂ε(a)

Properties:

Least Squares

Setting:

∂ε(a)

Properties:

Least Squares

Setting:

∂ε(a)

Properties:

Least Squares

Setting:

∂ε(a)

Properties:

Total Least Squares, TLS

Using SVD on matrix Es = W[E|s] we have:

Es = USVh ,

Solution:

aTLS = − 1

v2L+1,2L+1

[v0,2L+1 v1,2L+1 ... v2L,2L+1

]TErrors in matrix E are taken into accountSVD is computationally demandingand ... result is not satisfactory.

Es = USVh ,

Solution:

aTLS = − 1

v2L+1,2L+1

[v0,2L+1 v1,2L+1 ... v2L,2L+1

Es = USVh ,

Solution:

aTLS = − 1

v2L+1,2L+1

[v0,2L+1 v1,2L+1 ... v2L,2L+1

Es = USVh ,

Solution:

aTLS = − 1

v2L+1,2L+1

[v0,2L+1 v1,2L+1 ... v2L,2L+1

Es = USVh ,

Solution:

aTLS = − 1

v2L+1,2L+1

[v0,2L+1 v1,2L+1 ... v2L,2L+1

Non Linear Least Squares, NLS

Estimate both complex amplitudes and local fundamentalfrequency

Two iterative algorithms

Steepest DescentNewton-Gauss

Iteration step:

x(new) = x(old) − ηf ′(x(old)) ,

where η is the rate of correction.

The difference is in the iteration rate

NLS- Steepest Descent

Iteration step: [a(new)

f(new)

[a(old)

f(old)

]+ ηJe

where J is the Jacobian of the error vector e

Jacobian:

∂e[−N]∂a−L

∂e[−N+1]∂a−L

... ∂e[N]∂a−L

......

∂e[−N]∂aL

∂e[−N+1]∂aL

... ∂e[N]∂aL

∂e[−N]∂f0

∂e[−N+1]∂f0

... ∂e[N]∂f0

(2L+2×2N+1)

The rate, η, is manually adjusted.

f(new)

[a(old)

f(old)

]+ ηJe

Jacobian:

∂e[−N]∂a−L

∂e[−N+1]∂a−L

... ∂e[N]∂a−L

......

∂e[−N]∂aL

∂e[−N+1]∂aL

... ∂e[N]∂aL

∂e[−N]∂f0

∂e[−N+1]∂f0

... ∂e[N]∂f0

(2L+2×2N+1)

f(new)

[a(old)

f(old)

]+ ηJe

Jacobian:

∂e[−N]∂a−L

∂e[−N+1]∂a−L

... ∂e[N]∂a−L

......

∂e[−N]∂aL

∂e[−N+1]∂aL

... ∂e[N]∂aL

∂e[−N]∂f0

∂e[−N+1]∂f0

... ∂e[N]∂f0

(2L+2×2N+1)

NLS-Newton-Gauss

Iteration step:[a(new)

f(new)

[a(old)

f(old)

]+ (JhJ)−1Jhe

The correction rate, η, is adjusted by the inverse of the JhJ

Expression, (JhJ)−1Jhe, is similar to LS solution.

NLS-Newton-Gauss

f(new)

[a(old)

f(old)

]+ (JhJ)−1Jhe

NLS-Newton-Gauss

f(new)

[a(old)

f(old)

]+ (JhJ)−1Jhe

NLS - Initialization

First voiced frame: Through Least Squares

Subsequent frames: Using the parameters of the previousframe

NLS - Initialization

First voiced frame: Through Least Squares

Subsequent frames: Using the parameters of the previousframe

Comparisons

Average segmental SNR

LS TLS SD NG

26.70 26.70 26.95 28.40

Average number of iterations

17.3 3.8

Explanation on TLS

Whenf0 + ∆f0 = f0

then(E)N+n,L+l = e j2πl f0n

= e j2πlf0ne j2πl∆f0n

TLS expects an additive error, not a multiplicative

Again on the energy modulation

0 200 400 600 800 1000 1200 1400 1600 1800 2000−0.4

−0.2

Time in samples

0 200 400 600 800 1000 1200 1400 1600 1800 2000−0.1

−0.05

Time in samples

So far, mainly

So far we mainly use the Triangular Envelope:

Signal envelope

There are many ways to obtain the “envelope” of a signal, as:

Hilbert Transform (analytic signal)

Low-pass local energy (energy envelope):

e[n] =1

2N + 1

N∑k=−N

|r [n − k]|

where r [n] denotes the residual signal.

Signal envelope

There are many ways to obtain the “envelope” of a signal, as:

Hilbert Transform (analytic signal)

Low-pass local energy (energy envelope):

e[n] =1

2N + 1

N∑k=−N

|r [n − k]|

where r [n] denotes the residual signal.

Hilbert envelope

We may also use the Hilbert envelope, computed as:

eH [n] =L∑

k=L−M+1

ake2πk(f0/fs)n

Example of energy envelope

Example of Energy Envelope, with N = 7

0 200 400 600 800 1000 1200 1400 1600 1800 2000−0.1

−0.05

Time in samples

Energy envelope

The energy envelope can be efficiently parameterized with a fewFourier coefficients:

e[n] =Le∑

k=−Le

Akej2πk(f0/fs)n

where Le is set to be 3 to 4

Looking at time domain properties

200 400 600 800 1000 1200 1400 1600 1800 2000−0.1

Noise Part

200 400 600 800 1000 1200 1400 1600 1800 2000−0.1

Triangular Envelope

200 400 600 800 1000 1200 1400 1600 1800 2000−0.1

Hilbert Envelope

200 400 600 800 1000 1200 1400 1600 1800 2000−0.1

Energy [email protected] Harmonic plus Noise, HNM 52/59

Results from Listening test I

Triangular No pref. Hilbert

Male 8 (8.3%) 43 (44.8%) 45 (46.9%)

Female 40 (41.7%) 47 (48.9%) 9 (9.4%)

Hilbert No pref. Energy

Male 22 (22.9%) 47 (49.0%) 27 (28.1%)

Female 22 (22.9%) 54 (56.3%) 20 (20.8%)

Energy No pref. Triangular

Male 43 (44.8%) 50 (52.0%) 3 (3.2%)

Female 16 (16.7%) 67 (69.8%) 13 (13.5%)

Table: Results from the listening test for the English sentences.

Results from Listening test II

Triangular No pref. Hilbert

Male 10 (10.4%) 47 (49.0%) 39 (40.6%)

Female 8 (8.3%) 71 (74.0%) 17 (17.7%)

Hilbert No pref. Energy

Male 11 (11.5%) 58 (60.4%) 27 (28.1%)

Female 13 (13.5%) 58 (60.4%) 25 (26.1%)

Energy No pref. Triangular

Male 42 (43.7%) 48 (50.0%) 6 (6.3%)

Female 16 (16.7%) 68 (70.8%) 12 (12.5%)

Table: Results from the listening test for the French sentences.

Sound Examples using HNM1

Examples with HNM1 and Maximum Voiced frequency fixed at4000 Hz

Original Triangular Hilbert Energy

Female

Sound Examples using HNM2

Examples with HNM2 and Maximum Voiced frequency fixed at4000 Hz

Original Triangular Hilbert Energy

Female

Acknowledgments

My ex-PhD: Yannis Pantazis for his contributions to thefurther work on HNMs

THANK YOU

for your attention

D. Griffin and J. Lim, “Multiband-excitation vocoder,” IEEE Trans. Acoust., Speech, Signal Processing,

vol. ASSP-36, pp. 236–243, Fev 1988.

A. Abrantes, J. Marques, and I. Transcoso, “Hybrid sinusoidal modeling of speech without voicing decision,”

Eurospeech-91, pp. 231–234, 1991.

J. Laroche, Y. Stylianou, and E. Moulines, “HNS: Speech modification based on a harmonic + noise

model.,” Proc. IEEE ICASSP-93, Minneapolis, pp. 550–553, Apr 1993.

B.Yegnanarayana, C. d’Alessandro, and V. Darsinos, “An iterative algorithm for decomposition of speech

signals into periodic and aperiodic components,” vol. 6, no. 1, 1998.

Y. Stylianou, J. Laroche, and E. Moulines, “High-Quality Speech Modification based on a Harmonic +

Noise Model.,” Proc. EUROSPEECH, 1995.

Y. Stylianou, Harmonic plus Noise Models for Speech, combined with Statistical Methods, for Speech and

Speaker Modification.PhD thesis, Ecole Nationale Superieure des Telecommunications, Jan 1996.

W. Hess, Pitch determination of Speech Signals: Algorithmes and Devices.

Berlin: Springer, 1983.

I. Zeljkovic and Y. Stylianou, “Single complex sinusoid and arhe model based pitch extractors,” Proc.

EUROSPEECH, 1999.

Y. Stylianou, “Modeling speech based on harmonic plus noise models.,” in Nonlinear Speech Modeling and

Aplications (G. Chellot, A. Esposito, M. Faundez, and M. M, eds.), pp. 244–260, Springer-Verlag, 2005.

Harmonic plus Noise models

Documents

Transcript of Harmonic plus Noise models