Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal...
Transcript of Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal...
![Page 1: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/1.jpg)
Tutorial
Meinard Müller
Saarland University and MPI Informatik [email protected]
Music Signal Processing
Anssi Klapuri
Queen Mary University of London [email protected]
![Page 2: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/2.jpg)
Overview
Pitch and Harmony Tempo and Beat Timbre Melody
Part I:
Part II:
Part III:
Part IV:
Coffee Break
![Page 3: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/3.jpg)
Overview
Tempo and Beat
Part II:
§ Introduction § Onset detection § Tempo estimation § Beat/pulse estimation § Applications
![Page 4: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/4.jpg)
Introduction
Basic beat tracking task: Given an audio recording of a piece of music, determine the periodic sequence of beat positions. “Tapping the foot when listening to music’’
![Page 5: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/5.jpg)
Time (seconds)
Example: Queen – Another One Bites The Dust
Introduction
![Page 6: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/6.jpg)
Example: Queen – Another One Bites The Dust
Time (seconds)
Introduction
![Page 7: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/7.jpg)
Introduction
Example: Happy Birthday to you
Pulse level: Measure
![Page 8: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/8.jpg)
Introduction
Example: Happy Birthday to you
Pulse level: Tactus (beat)
![Page 9: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/9.jpg)
Introduction
Example: Happy Birthday to you
Pulse level: Tatum (temporal atom)
![Page 10: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/10.jpg)
Example: Chopin – Mazurka Op. 68-3
Pulse level: Quarter note
Tempo: ???
Introduction
![Page 11: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/11.jpg)
Example: Chopin – Mazurka Op. 68-3
Pulse level: Quarter note
Tempo: 50-200 BPM
Time (beats)
Tem
po (B
PM
)
50
200
Tempo curve
Introduction
![Page 12: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/12.jpg)
Introduction
Example: Borodin – String Quartet No. 2
Pulse level: Quarter note
Tempo: 120-140 BPM (roughly)
Beat tracker without any prior knowledge
Beat tracker with prior knowledge on rough tempo range
![Page 13: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/13.jpg)
Introduction
§ Pulse level often unclear
§ Local/sudden tempo changes (e.g. rubato)
§ Vague information
(e.g., soft onsets, extracted onsets corrupt)
§ Sparse information
(often only note onsets are used)
Challenges in beat tracking
![Page 14: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/14.jpg)
§ Onset detection § Beat tracking § Tempo estimation
Tasks
Introduction
![Page 15: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/15.jpg)
§ Onset detection § Beat tracking § Tempo estimation
Tasks
Introduction
![Page 16: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/16.jpg)
period phase
§ Onset detection § Beat tracking § Tempo estimation
Tasks
Introduction
![Page 17: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/17.jpg)
Tempo := 60 / period Beats per minute (BPM)
§ Onset detection § Beat tracking § Tempo estimation
Tasks
period
Introduction
![Page 18: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/18.jpg)
Onset Detection
§ Finding start times of perceptually relevant acoustic events in music signal
§ Onset is the time position where a note is played
§ Onset typically goes along
with a change of the signal’s properties: – energy or loudness – pitch or harmony – timbre
![Page 19: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/19.jpg)
Onset Detection
[Bello et al., IEEE-TASLP 2005]
§ Finding start times of perceptually relevant acoustic events in music signal
§ Onset is the time position where a note is played
§ Onset typically goes along
with a change of the signal’s properties: – energy or loudness – pitch or harmony – timbre
![Page 20: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/20.jpg)
Steps
Time (seconds)
Onset Detection (Energy-Based)
Waveform
![Page 21: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/21.jpg)
Onset Detection (Energy-Based)
Time (seconds)
Squared waveform
Steps 1. Amplitude squaring
![Page 22: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/22.jpg)
Onset Detection (Energy-Based)
Time (seconds)
Energy envelope
Steps 1. Amplitude squaring 2. Windowing
![Page 23: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/23.jpg)
Onset Detection (Energy-Based)
Capturing energy changes
Time (seconds)
Differentiated energy envelope
Steps 1. Amplitude squaring 2. Windowing 3. Differentiation
![Page 24: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/24.jpg)
Onset Detection (Energy-Based)
Time (seconds)
Novelty curve
Steps 1. Amplitude squaring 2. Windowing 3. Differentiation 4. Half wave rectification
Only energy increases are relevant for note onsets
![Page 25: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/25.jpg)
Onset Detection (Energy-Based)
Time (seconds)
Steps 1. Amplitude squaring 2. Windowing 3. Differentiation 4. Half wave rectification 5. Peak picking
Peak positions indicate note onset candidates
![Page 26: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/26.jpg)
Energy envelope
Onset Detection (Energy-Based)
Time (seconds)
![Page 27: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/27.jpg)
Onset Detection (Energy-Based)
Time (seconds)
Energy envelope / note onsets positions
![Page 28: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/28.jpg)
Onset Detection
§ Energy curves often only work for percussive music
§ Many instruments such as strings have weak note onsets
§ No energy increase may be observable in complex sound mixtures
§ More refined methods needed that capture – changes of spectral content – changes of pitch – changes of harmony
[Bello et al., IEEE-TASLP 2005]
![Page 29: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/29.jpg)
1. Spectrogram Magnitude spectrogram
Freq
uenc
y (H
z)
Time (seconds)
|| X Steps: Onset Detection (Spectral-Based)
§ Aspects concerning pitch, harmony, or timbre are captured by spectrogram
§ Allows for detecting local energy changes in certain frequency ranges
![Page 30: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/30.jpg)
Compressed spectrogram Y
|)|1log( XCY ⋅+=
Onset Detection (Spectral-Based)
1. Spectrogram 2. Logarithmic compression
Steps:
§ Accounts for the logarithmic sensation of sound intensity
§ Dynamic range compression § Enhancement of low-intensity
values § Often leading to enhancement
of high-frequency spectrum
Time (seconds)
Freq
uenc
y (H
z)
![Page 31: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/31.jpg)
Spectral difference
Onset Detection (Spectral-Based)
1. Spectrogram 2. Logarithmic compression 3. Differentiation
Steps:
§ First-order temporal difference
§ Captures changes of the spectral content
§ Only positive intensity changes considered
Time (seconds)
Freq
uenc
y (H
z)
![Page 32: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/32.jpg)
Spectral difference
t Novelty curve
Onset Detection (Spectral-Based)
1. Spectrogram 2. Logarithmic compression 3. Differentiation 4. Accumulation
Steps:
§ Frame-wise accumulation of all positive intensity changes
§ Encodes changes of the spectral content
Freq
uenc
y (H
z)
![Page 33: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/33.jpg)
Onset Detection (Spectral-Based)
1. Spectrogram 2. Logarithmic compression 3. Differentiation 4. Accumulation
Steps:
Novelty curve
![Page 34: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/34.jpg)
Substraction of local average
Onset Detection (Spectral-Based)
1. Spectrogram 2. Logarithmic compression 3. Differentiation 4. Accumulation 5. Normalization
Steps:
Novelty curve
![Page 35: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/35.jpg)
Onset Detection (Spectral-Based)
1. Spectrogram 2. Logarithmic compression 3. Differentiation 4. Accumulation 5. Normalization
Steps:
Normalized novelty curve
![Page 36: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/36.jpg)
Onset Detection (Spectral-Based)
1. Spectrogram 2. Logarithmic compression 3. Differentiation 4. Accumulation 5. Normalization 6. Peak picking
Steps:
Normalized novelty curve
![Page 37: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/37.jpg)
Logarithmic compression is essential
Novelty curve
Onset Detection (Spectral-Based)
Time (seconds)
Freq
uenc
y (H
z)
|| X
Ground-truth onsets
[Klapuri et al., IEEE-TASLP 2006]
![Page 38: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/38.jpg)
C = 1
Onset Detection (Spectral-Based)
|)|1log( XCY ⋅+=
Logarithmic compression is essential
Novelty curve
Ground-truth onsets
[Klapuri et al., IEEE-TASLP 2006]
Freq
uenc
y (H
z)
Time (seconds)
![Page 39: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/39.jpg)
Onset Detection (Spectral-Based) Logarithmic compression is essential
Novelty curve
Ground-truth onsets
C = 10
|)|1log( XCY ⋅+=
[Klapuri et al., IEEE-TASLP 2006]
Freq
uenc
y (H
z)
Time (seconds)
![Page 40: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/40.jpg)
Onset Detection (Spectral-Based) Logarithmic compression is essential
Novelty curve
Ground-truth onsets
C = 1000
|)|1log( XCY ⋅+=
[Klapuri et al., IEEE-TASLP 2006]
Freq
uenc
y (H
z)
Time (seconds)
![Page 41: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/41.jpg)
Time (seconds)
Onset Detection
Peak picking
§ Peaks of the novelty curve indicate note onset candidates
![Page 42: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/42.jpg)
Time (seconds)
Onset Detection
Peak picking
§ Peaks of the novelty curve indicate note onset candidates § In general many spurious peaks § Usage of local thresholding techniques § Peak-picking very fragile step in particular for soft onsets
![Page 43: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/43.jpg)
Onset Detection
Shostakovich – 2nd Waltz
Time (seconds)
Time (seconds)
Borodin – String Quartet No. 2
![Page 44: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/44.jpg)
Beat and Tempo
§ Steady pulse that drives music forward and provides the temporal framework of a piece of music
§ Sequence of perceived pulses that are equally spaced in time
§ The pulse a human taps along when listening to the music
[Parncutt 1994]
[Sethares 2007]
[Large/Palmer 2002]
[Lerdahl/ Jackendoff 1983]
[Fitch/ Rosenfeld 2007]
What is a beat?
The term tempo then refers to the speed of the pulse.
![Page 45: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/45.jpg)
Beat and Tempo
§ Analyze the novelty curve with respect to reoccurring or quasi-periodic patterns
§ Avoid the explicit determination of note onsets (no peak picking)
Strategy
![Page 46: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/46.jpg)
Beat and Tempo
[Scheirer, JASA 1998]
[Ellis, JNMR 2007]
[Davies/Plumbley, IEEE-TASLP 2007]
[Peeters, JASP 2007]
Strategy
§ Comb-filter methods § Autocorrelation § Fourier transfrom
Methods
[Grosche/Müller, ISMIR 2009]
§ Analyze the novelty curve with respect to reoccurring or quasi-periodic patterns
§ Avoid the explicit determination of note onsets (no peak picking)
![Page 47: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/47.jpg)
Definition: A tempogram is a time-tempo representation that encodes the local tempo of a music signal over time.
Tem
po (B
PM
)
Time (seconds)
Inte
nsity
Tempogram
![Page 48: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/48.jpg)
Definition: A tempogram is a time-tempo represenation that encodes the local tempo of a music signal over time.
§ Compute a spectrogram (STFT) of the novelty curve § Convert frequency axis (given in Hertz) into
tempo axis (given in BPM) § Magnitude spectrogram indicates local tempo
Fourier-based method
Tempogram (Fourier)
![Page 49: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/49.jpg)
Tem
po (B
PM
)
Time (seconds)
Tempogram (Fourier)
Novelty curve
![Page 50: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/50.jpg)
Tem
po (B
PM
)
Time (seconds)
Tempogram (Fourier)
Novelty curve (local section)
![Page 51: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/51.jpg)
Tem
po (B
PM
)
Windowed sinusoidal
Time (seconds)
Tempogram (Fourier)
![Page 52: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/52.jpg)
Tem
po (B
PM
)
Windowed sinusoidal
Time (seconds)
Tempogram (Fourier)
![Page 53: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/53.jpg)
Tem
po (B
PM
)
Time (seconds)
Tempogram (Fourier)
Windowed sinusoidal
![Page 54: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/54.jpg)
Definition: A tempogram is a time-tempo represenation that encodes the local tempo of a music signal over time.
§ Compare novelty curve with time-lagged local sections of itself
§ Convert lag-axis (given in seconds) into tempo axis (given in BPM)
§ Autocorrelogram indicates local tempo
Autocorrelation-based method
Tempogram (Autocorrelation)
![Page 55: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/55.jpg)
Tempogram (Autocorrelation)
Novelty curve (local section)
Lag
(sec
onds
)
Time (seconds)
![Page 56: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/56.jpg)
Tempogram (Autocorrelation) La
g (s
econ
ds)
Windowed autocorrelation
![Page 57: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/57.jpg)
Tempogram (Autocorrelation)
Lag = 0 (seconds)
Lag
(sec
onds
)
![Page 58: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/58.jpg)
Tempogram (Autocorrelation) La
g (s
econ
ds)
Lag = 0.26 (seconds)
![Page 59: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/59.jpg)
Tempogram (Autocorrelation) La
g (s
econ
ds)
Lag = 0.52 (seconds)
![Page 60: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/60.jpg)
Tempogram (Autocorrelation)
Lag = 0.78 (seconds)
Lag
(sec
onds
)
![Page 61: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/61.jpg)
Tempogram (Autocorrelation) La
g (s
econ
ds)
Lag = 1.56 (seconds)
![Page 62: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/62.jpg)
Lag
(sec
onds
) Tempogram (Autocorrelation)
Time (seconds)
Time (seconds)
![Page 63: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/63.jpg)
Time (seconds)
300
60 80
40
30
120
Tempogram (Autocorrelation) Te
mpo
(BP
M)
Time (seconds)
![Page 64: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/64.jpg)
600
500
400
300
200
100
Tempogram (Autocorrelation)
Time (seconds)
Tem
po (B
PM
)
Time (seconds)
![Page 65: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/65.jpg)
Time (seconds)
Tempogram Fourier Autocorrelation
Time (seconds)
Tem
po (B
PM
)
![Page 66: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/66.jpg)
Time (seconds)
Tempogram Fourier Autocorrelation
Time (seconds)
210
70
Tem
po (B
PM
)
Tempo@Tatum = 210 BPM Tempo@Measure = 70 BPM
![Page 67: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/67.jpg)
Tempogram
Fourier Autocorrelation
Time (seconds) Time (seconds)
Tem
po (B
PM
)
Time (seconds)
Emphasis of tempo harmonics (integer multiples)
Emphasis of tempo subharmonics (integer fractions)
[Grosche et al., ICASSP 2010] [Peeters, JASP 2007]
![Page 68: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/68.jpg)
Tempogram (Summary)
Fourier Autocorrelation
Novelty curve is compared with sinusoidal kernels each representing a specific tempo
Novelty curve is compared with time-lagged local (windowed) sections of itself
Convert frequency (Hertz) into tempo (BPM)
Convert time-lag (seconds) into tempo (BPM)
Reveals novelty periodicities Reveals novelty self-similarities
Emphasizes harmonics Emphasizes subharmonics
Suitable to analyze tempo on tatum and tactus level
Suitable to analyze tempo on tatum and measure level
![Page 69: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/69.jpg)
Beat Tracking
§ Given the tempo, find the best sequence of beats
§ Complex Fourier tempogram contains magnitude and phase information
§ The magnitude encodes how well the novelty curve
resonates with a sinusoidal kernel of a specific tempo
§ The phase optimally aligns the sinusoidal kernel with the peaks of the novelty curve
[Peeters, JASP 2005]
![Page 70: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/70.jpg)
Beat Tracking Te
mpo
(BP
M)
Inte
nsity
[Peeters, JASP 2005]
![Page 71: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/71.jpg)
Beat Tracking Te
mpo
(BP
M)
Inte
nsity
[Peeters, JASP 2005]
![Page 72: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/72.jpg)
Beat Tracking Te
mpo
(BP
M)
Inte
nsity
[Peeters, JASP 2005]
![Page 73: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/73.jpg)
Beat Tracking Te
mpo
(BP
M)
Inte
nsity
![Page 74: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/74.jpg)
Tem
po (B
PM
)
Inte
nsity
Time (seconds)
Beat Tracking
[Grosche/Müller, IEEE-TASLP 2011]
![Page 75: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/75.jpg)
Beat Tracking
Novelty Curve
Predominant Local Pulse (PLP)
Time (seconds) [Grosche/Müller, IEEE-TASLP 2011]
![Page 76: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/76.jpg)
§ Periodicity enhancement of novelty curve § Accumulation introduces error robustness § Locality of kernels handles tempo variations
§ Indicates note onset candidates § Extraction errors in particular for soft onsets § Simple peak-picking problematic
Beat Tracking
Predominant Local Pulse (PLP)
Novelty Curve
[Grosche/Müller, IEEE-TASLP 2011]
![Page 77: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/77.jpg)
Beat Tracking Te
mpo
(BP
M)
Time (seconds)
Borodin – String Quartet No. 2
[Grosche/Müller, IEEE-TASLP 2011]
![Page 78: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/78.jpg)
Beat Tracking
Time (seconds)
Tem
po (B
PM
)
Borodin – String Quartet No. 2
[Grosche/Müller, IEEE-TASLP 2011]
Strategy: Exploit additional knowledge (e.g. rough tempo range)
![Page 79: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/79.jpg)
Applications
§ Feature design (beat-synchronous features, adaptive windowing)
§ Digital DJ / audio editing (mixing and blending of audio material)
§ Music classification § Music recommendation § Performance analysis
(extraction of tempo curves)
![Page 80: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/80.jpg)
Application: Feature Design
Fixed window size
[Ellis et al., ICASSP 2008] [Bello/Pickens, ISMIR 2005]
![Page 81: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/81.jpg)
[Bello/Pickens, ISMIR 2005]
Application: Feature Design
Fixed window size Adaptive window size
[Ellis et al., ICASSP 2008]
![Page 82: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/82.jpg)
Application: Feature Design
Fixed window size (100 ms)
Time (seconds)
![Page 83: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/83.jpg)
Application: Feature Design
Adative window size (roughly 1200 ms) Note onset positions define boundaries
Time (seconds)
![Page 84: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/84.jpg)
Application: Feature Design
Time (seconds)
Denoising by excluding boundary neighborhoods
Adative window size (roughly 1200 ms) Note onset positions define boundaries
![Page 85: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/85.jpg)
Application: Audio Editing (Digital DJ)
http://www.mixxx.org/
![Page 86: Music Signal Processing - ECE/CISarce/Courses_files/Eleg633-DImaging-14.… · Music Signal Processing ... Another One Bites The Dust Introduction . Example: Queen ... Borodin –](https://reader033.fdocuments.net/reader033/viewer/2022052709/5a78d3b47f8b9a83238d02ce/html5/thumbnails/86.jpg)
Application: Beat-Synchronous Light Effects