Time%frequency-Masking- - Interactive Audio Lab ·...

Time-‐frequency Masking

EECS 352: Machine Percep=on of Music & Audio

Zafar Rafii, Winter 2014 1

Real spectrogram

•  The Short-‐Time Fourier Transform (STFT) is a succession of local Fourier Transforms (FT)

+ j* Imaginary spectrogram

frame i frame i

Time signal

timewindow i

Imaginary spectrum

frequency

Real spectrum

frequency

•  If we used a window of N samples, the FT has N values, from 0 to N-‐1; e.g., if N = 8…

Time signal

timewindow i

+ j* window i frame i frame i

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

N frequency values N frequency values

Imaginary spectrum

frequency

Real spectrum

frequency

•  Frequency index 0 is the DC component; it is always real (it is the sum of the =me values!)

Time signal

timewindow i

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Imaginary spectrum

frequency

Real spectrum

frequency

•  Frequency indices from 1 to floor(N/2) are the “unique” complex values (a + j*b)

Time signal

timewindow i

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Imaginary spectrum

frequency

Real spectrum

frequency

•  Frequency indices from floor(N/2) to N-‐1 are the “mirrored” complex conjugates (a -‐ j*b)

Time signal

timewindow i

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Imaginary spectrum

frequency

Real spectrum

frequency

•  If N is even, there is a pivot component at frequency index N/2; it is always real!

Time signal

timewindow i

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Real spectrogram

•  Summary of the frequency indices and values in the STFT (in colors!)

Imaginary spectrogram

Frequency 0 = DC component (always real)

Frequency 1 to floor(N/2) = “unique” complex values

Frequency N/2 = “pivot” component (always real)

Frequency floor(N/2) to N-‐1 = “mirrored” complex conjugates

N frequency values = frequency 0 to N-‐1

•  The (magnitude) spectrogram is the magnitude (absolute value) of the STFT

Magnitude spectrogram

Real spectrogram

Spectrogram

frame i frame i frame i

•  For a complex number 𝑎+𝑗∗𝑏, the absolute value is |𝑎+𝑗∗𝑏|=√𝑎↑2 + 𝑏↑2  

Spectrogram

Imaginary spectrum

frequency

Real spectrum

frequency

+ j* frame i frame i

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Magnitude spectrum

frequency

frame i

1.4 1.4 1.4 1.4 3 0 2 0

1.4 1.4 1.4

•  All the N frequency values (frequency indices from 0 to N-‐1) are real and posiHve (abs!)

Spectrogram

Imaginary spectrum

frequency

Real spectrum

frequency

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Magnitude spectrum

frequency

0 1 2 3 4 5 6 7 = 1.4 3 0 2 0

frame i

N frequency values

1.4 1.4 1.4

•  Frequency indices from 0 to floor(N/2) are the unique frequency values (with DC and pivot)

Spectrogram

Imaginary spectrum

frequency

Real spectrum

frequency

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Magnitude spectrum

frequency

0 1 2 3 4 5 6 7 = 1.4 3 0 2 0

frame i

1.4 1.4 1.4

•  Frequency indices from floor(N/2)+1 to N-‐1 are the mirrored frequency values

Spectrogram

Imaginary spectrum

frequency

Real spectrum

frequency

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Magnitude spectrum

frequency

0 1 2 3 4 5 6 7 = 1.4 3 0 2 0

frame i

1.4 1.4 1.4

•  Since they are redundant, we can discard the frequency values from floor(N/2)+1 to N-‐1

Spectrogram

Imaginary spectrum

frequency

Real spectrum

frequency

-‐3 -‐1 0 1 2 1 0 -‐1 0 -‐1 0 1 0 -‐1 0 1 + j* 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7

Magnitude spectrum

frequency

0 1 2 3 4 5 6 7 = 1.4 3 0 2 0

frame i

floor(N/2)+1 unique frequency values

•  The spectrogram has therefore floor(N/2)+1 unique frequency values (with DC and pivot)

Real spectrogram

Spectrogram

frame i frame i frame i

abs + j*

Spectrogram

•  Why the magnitude spectrogram? – Easy to visualize (compare with the STFT) – Magnitude informa=on more important – Human ear less sensi=ve to phase

Time signal

ency +

Spectrogram

•  When you display a spectrogram in Matlab… –  imagesc: data is scaled to use the full colormap – 10*log10(V): magnitude spectrogram in dB – set(gca,’YDir’,’normal’): y-‐axis from boiom to top

Time signal

ency +

•  The signal cannot be reconstructed from the spectrogram (phase informa=on is missing!)

Spectrogram

Time signal

Real spectrogram

ency iSTFT ???

•  Suppose we have a mixture of two sources: a music signal and a voice signal

Music signal

Mixture signal

Voice signal

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

•  We assume that the sources are sparse = most of the =me-‐frequency bins have null energy

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Mixture signal

Voice signal

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

•  We assume that the sources are sparse = most of the =me-‐frequency bins have null energy

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Mixture signal

Voice signal

Music spectrogram

ency +

Voice spectrogram

Mostly low energy bins

Mostly low energy bins Mixture spectrogram

timefre

•  We assume that the sources are disjoint = their =me-‐frequency bins do not overlap

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Mixture signal

Voice signal

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

•  We assume that the sources are disjoint = their =me-‐frequency bins do not overlap

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Mixture signal

Voice signal

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

Not a lot of overlapping

•  Assuming sparseness and disjointness, we can discriminate the bins between mixed sources

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Mixture signal

Voice signal

•  Assuming sparseness and disjointness, we can discriminate the bins between mixed sources

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Mixture signal

Voice signal

Source 1 = bright Source 2 = dark

•  Bins that are likely to belong to one source are assigned to 1, the rest to 0 = binary masking!

Binary mask

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

Binary mask

ency 1

Music signal

Voice signal

Source of interest Interfering source

•  By mul=plying the binary mask to the mixture spectrogram, we can “preview” the es=mate

Binary mask

ency .x

Mixture spectrogram

Masked spectrogram

•  However, we cannot derive the es=mate itself because we cannot invert a spectrogram!

Binary mask

ency .x

Mixture spectrogram

Masked spectrogram

Music estimate

•  We mirror the redundant frequencies from the unique frequencies (without DC and pivot)

Binary mask

frequency

•  We then apply this full binary mask to the STFT using a element-‐wise mul=plica=on

Binary mask

frequency

Real spectrogram

timefre

•  The es=mate signal can now be reconstructed via inverse STFT

Binary mask

Masked imaginary

Masked real

Music estimate

•  Sources are not really sparse or disjoint in =me-‐frequency in the mixture

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Mixture signal

Voice signal

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

Soft mask

•  Bins that are likely to belong to one source are close to 1, the rest close to 0 = soP masking!

Music spectrogram

Voice spectrogram

Mixture spectrogram

timefre

Music spectrogram

ency +

Voice spectrogram

Mixture spectrogram

timefre

Music signal

Voice signal

Source of interest Interfering source

Soft mask

•  Let’s listen to the results!

Music estimate

Music signal

demix mix

Mixture signal

•  How can we efficiently model a binary/som =me-‐frequency mask for source separa=on?...

•  To be con=nued…

Ques=on

Mixture spectrogram

Soft mask

1 0 ???

Time%frequency-Masking- - Interactive Audio Lab ·...

Documents

Transcript of Time%frequency-Masking- - Interactive Audio Lab ·...

Time frequency localization

Time–Frequency Analysis of Small Frequency Variations in ...

DataVu-PC Analysis Software€¦ · Spectrogram Scale Enable or disable the Spectrogram view. Set the Maximum and Minimum Power for the Spectrogram view. Power v. Time Scale Enable

Sparse Frequency Extrapolation of Spectrograms...the Fourier domain. Plotting the magnitude of the transform as a function of time provides a spectrogram on how the frequencies of

Signal Reconstruction from its Spectrogram

Time-frequency analysis - Aalto · Time-frequency analysis Dr. Ville Turunen ... apply sharp time-frequency operations to signals, ... A chosen Cohen class time-frequency transform

0764) Issue 02, March 2020 Continuous Speech and Time ...time-frequency transform is also known as time-frequency analysis. The graph of the time-frequency analysis is called a spectrogram.

Spectrogram Time-Frequency Analysis and Classification of ... · location of transmitters, estimation of modulation parameters and ... classification accuracy is 90 percent for signal-to-noise

Music Processing Applications of Music Processing · 2017. 2. 14. · Gabor Wavelet Spectrogram [dB] Time [Sec] Frequency [Hz] 0.5 1 1.5 2 277 407 599 880 1293 1901 2794 4106 6035

Spectrogram Examples - Maplesoft Spectrogram Examples Introduction A spectrogram illustrates how the constituent frequencies of a signal vary over time. This ... by the Nine Inch Nails;

Learning&phonology& Learning&Phonology& Spectrogram&

Time frequency analysis_journey

Spectrogram & its reading by Tae-Yeoub Jang. Reviving Sonus 2 What is spectrogram? Begin to be used since 1940s Another representation of frequency domain.

TIME-FREQUENCY ANALYSIS - INAF - OA-Brerabelloni/ASTROSAT/Home_files/Time-Frequency.pdf · TIME-FREQUENCY ANALYSIS ... COHEN’S KERNEL ALL time-frequency representations come from:

Circuits for Time and Frequency Domain …digitool.library.mcgill.ca/thesisfile29532.pdfCircuits for Time and Frequency Domain Characterization ... Time and frequency domain characterization

mentor.ieee.org€¦ · Web viewIt is known that the characteristic of frequency band is related to cybersickness. In order to consider the frequency characteristics, spectrogram

Spectrogram using short-time Fourier transform - MATLAB spectrogram.pdf

Spectrogram Segmentation 2

Time-frequency Analysis of Musical · PDF file‘–gas a ﬁnite ... piano scale. (c) Spectrogram of a piano solo from a Bach melody. ... sines and cosines used in classical Fourier

TIME-FREQUENCY REPRESENTATIONS OF EARTHQUAKE MOTION … · time-frequency representations of earthquake motion records ... time-frequency representations of earthquake motion ...