Shaft Alignment: Calculations for Horizontal Shaft Alignment
Methods and Datasets for DJ-Mix Reverse ... - fourer.fr · Step 1: Rough Alignment by DTW Dynamic...
18
Methods and Datasets for DJ-Mix Reverse Engineering Diemo Schwarz, Dominique Fourer Ircam Lab, CNRS, Sorbonne Université, Ministère de la Culture, Paris, France IBISC, Université d’Évry-Val-d’Essonne/Paris-Saclay, Évry, France ABC_DJ Artist to Business to Business to Consumer Audio Branding System http://zenodo.org/record/1422385, http://www.ircam.fr, http://abcdj.eu
Transcript of Methods and Datasets for DJ-Mix Reverse ... - fourer.fr · Step 1: Rough Alignment by DTW Dynamic...
Methods and Datasets for DJ-Mix Reverse EngineeringDiemo Schwarz, Dominique FourerIrcam Lab, CNRS, Sorbonne Université, Ministère de la Culture, Paris, FranceIBISC, Université d’Évry-Val-d’Essonne/Paris-Saclay, Évry, France
ABC_DJ Artist to Business to Business to Consumer Audio Branding System
http://zenodo.org/record/1422385, http://www.ircam.fr, http://abcdj.eu
Collaboration Context: The ABC DJ EU-Project
MIR tools for audio brandingautomatic, DJ-like playback of playlists in stores
ABC_DJ Artist to Business to Business to Consumer Audio Branding System
http://abcdj.eu
The ABC DJ project has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement No 688122.
HearDis! GmbH
Scientific Context: Understanding DJ Culture & Practices
Important part of popular music culture
Enables: - musicological research in popular music - studies on DJ culture- computer support of DJing- automation of DJ mixing
Qualitative accounts exists, but…
Problem: Lack of Annotated Databases of DJ Mixes or DJ Sets
Very large scale availability (millions) of DJ mixes, often with tracklist, e.g. http://www.mixcloud.com, YouTube, podcasts.
very few annotated databases
Existing research in studio multi-track mixing and unmixing in DAWs
Existing work on DJ production tools, but no information retrieval from recorded mixes
Needed Components
Identification
DJ Mixes
Alignment
Unmixing
Time-Scaling
Mix Data
Audio Tracks
Playlistget track start and end in mix determine tempo changes (beat-aligned mixing) suggested here
identify contained tracks (fingerprinting)
estimate fade curves for volume, bass/treble, and parameters of other effects (compression, echo, etc.)
derive genre and social tags attached to the music → inform about the choices a DJ makes when creating a mix
downstream research enabled by DJ mix annotation
Cultural Data
Content Analysis
Context Analysis
Proposed Method for DJ Mix Reverse Engineering
Input • recorded DJ mix• playlist (list of tracks in the mix in correct order)• audio files of the original tracks
Five steps 1.rough alignment2.sample alignment3.verification by track removal4.estimation of gain curves5.estimation of cue regions
Step 1: Rough Alignment by DTW
Dynamic Time Warping alignment of concatenated MFCCs of tracks with mix → relative positioning of the tracks in
the mix (intersections)→ speed factor (slopes of path)
track frames
2000 4000 6000 8000 10000 12000
mix
fra
mes
1000
2000
3000
4000
5000
6000
7000
-50
-40
-30
-20
-10
0
Query index
Ref
eren
ce in
dex
-2 -1 0
-2-1
0
mix
MFC
Cs
concatenated tracks’ MFCCs
Step 2: Sample Alignment
Refine alignment to close in to sample precision:
1. time-scale source track according to estimated speed factor
2. search best sample shift around rough (frame) alignment maximum cross-correlation between mix and track
Step 3: Verification by Track Removal
Success of sample alignment can be verified by subtracting the aligned and time-scaled track from the mix → drop in RMS energy/!\ Method applicable even when ground truth is unknown or inexact!
-10
0
10
set123mix3-none-bass-04
-10
0
10
seconds
0 15 30 45 60 75 90 105
-10
0
10
diffe
renc
e of
RM
S en
ergy
[dB]
Step 4: Volume Curve Estimation
Estimate the volume curves âi (black lines) applied to each track to obtain the mixNovel method based on time-frequency representations X (mix) and Si (track):
6 Diemo Schwarz and Dominique Fourer
1. we compute the short-time Fourier transforms (STFT) of x and si denotedSi(n,m) and X(n,m) (n and m being respectively the time and frequencyindices)
2. we estimate the volume curve at each instant n by computing the medianof the mix/track ratio computed along the frequencies m0 2 M, where M isthe set of frequency indices where |Si(n,m0)|2 > 0, such as:
ai(n)=
8<
:median
⇣|X(n,m0
)||Si(n,m0
)|
⌘
8m02Mif 9m0 s. t. |Si(n,m0)|2 > 0
0 otherwise(2)
3. we optionally post-process ai(n) to obtain a smooth curve by removing out-liers using a second median filter for which a kernel size equal to 20 providesgood results in practice.
The resulting volume curve can then be used to estimate the cue points(the time instants when a fading e↵ect begins or stops) at the next step. Anillustration of the resulting process is presented in Fig. 3.
3.5 Step 5: Cue Point Estimation
In order to estimate the DJ cue points, we apply a linear regression of ai atthe time instants located at the beginning and at the end of the resultingvolume curve (when ai(n) < � , � being a threshold defined arbitrarily as� = 0.7max(a)). Assuming that a linear fading e↵ect was applied, the cuepoints can easily be deduced from the two a�ne equations resulting from thelinear regression. The four estimated cue points correspond respectively to:
1. n
1
, the time instant when the fade-in curve is equal to 02. n
2
, the time instant when the fade-in curve is equal to max(ai)3. n
3
, the time instant when the fade-out curve is equal to max(ai)4. n
4
, the time instant when the fade-out curve is equal to 0.
In order to illustrate the e�ciency of the entire method (steps 4 and 5), wepresent in Fig. 3 the results obtained on a real-world DJ-mix extracted from ourproposed dataset.
Fig. 3: Estimated volume curve (black), linear fades (blue), ground truth fades (red)
Step 5: Cue Point Estimation
Cue points are the start and end points of fadesEstimation (blue lines) by linear regression of the fade curve â at beginning and end (where â is between 0 and 70% of its maximum)Ground truth fade curve in red
The UnmixDB Open DJ-Mix Dataset
Automatically generated “ecologically valid” beat-synchronous mixes based on CC-licensed freely available music tracks from net label http://www.mixotic.net curated by Sonnleitner, Arzt & Widmer (2016)Each mix combines 3 track excerpts of ~40s (start cutting into end on a downbeat)Precise ground truth about the placement of tracks in a mix, fade curves, speedMixes generated in 12 variants:
4 effects: no effect, bass boost, dynamics compression, distortion 3 time-scaling algorithms: none, resample, time stretch
6 sets of tracks and mixes, 500 MB – 1 GB, total 4 GBpython source code for mix generation at https://github.com/Ircam-RnD/unmixdb-creation
http://zenodo.org/record/1422385
track frames
2000 4000 6000 8000 10000 12000
mix
fra
me
s
1000
2000
3000
4000
5000
6000
7000
-50
-40
-30
-20
-10
0
Evaluation Measures and Results: Alignment
frame error: absolute error between ground truth and frame start time from DTW rough alignment (step 1) [s]
sample error: absolute error between ground truth and track start time from sample alignment (step 2) [s]
log f
ram
ee
rro
r [s
]
10-4
10-3
10-2
10-1
100
101
102
103
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
log s
am
ple
err
or
[s]
10-4
10-3
10-2
10-1
100
101
102
103
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
no stretch resample stretch
no fx bass compression distortion
Evaluation Measures and Results: Speed Ratio
speed ratio: ratio between ground truth and speed factor estimated by DTW alignment (step 1, ideal value is 1)
speedra
tio0.96
0.97
0.98
0.99
1
1.01
1.02
1.03
1.04
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
track frames
2000 4000 6000 8000 10000 12000
mix
fra
me
s
1000
2000
3000
4000
5000
6000
7000
-50
-40
-30
-20
-10
0
Reinjecting Ground Truth Speed
High sensitivity of sample alignment and track removal on accuracy of speed estimation from DTWJudge its influence by reinjecting ground truth speed:
top: sample alignment error
bottom: sample alignment error with ground truth speed reinjected for time-scaling
log r
isam
ple
err
or
[s]
10-4
10-3
10-2
10-1
100
101
102
103
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
log
sa
mp
lee
rro
r [s
]
10-4
10-3
10-2
10-1
100
101
102
103
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
-10
0
10
set123mix3-none-bass-04
-10
0
10
seconds
0 15 30 45 60 75 90 105
-10
0
10
Evaluation Measures and Results: Suppression
suppression ratio: ratio of track time with >15 dB of track removal (step 3, bigger is better)
top: with DTW-estimated speed
bottom: with ground truth speed for time-scaling
suppra
tio
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
risu
pp
ratio
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
Evaluation Measures and Results: Fade Error
fade error: total difference between ground truth and estimated fade curves (steps 4 and 5)
fade
err
or
[db
/s]
0
10
20
30
40
50
60
70
80
90
100
none none none none resamp resamp resamp resamp stretch stretch stretch stretchnone bass comp dist none bass comp dist none bass comp dist
Conclusion and Future Work
Our DJ-mix reverse engineering method validated on artificial open UnmixDB dataset → retrieval of rich data from existing real DJ mixes
With some refinements, our method could become robust and precise enough to allow the inversion of EQ and other processing (e.g. compression, echo)
Extend to mixes with non-constant tempo curves, more effects
Close link between alignment, time-scaling, and unmixing hints at a joint and possibly iterative estimation algorithm
Other signal representations (MFCC, spectrum, chroma, scattering transform)?
beware: IANADJ!
