Automatic Transcription of Drums and Vocalised percussion · 2019. 7. 14. · segue o método...
Transcript of Automatic Transcription of Drums and Vocalised percussion · 2019. 7. 14. · segue o método...
FACULDADE DE ENGENHARIA DA UNIVERSIDADE DO PORTO
Automatic Transcription of Drums andVocalised percussion
António Filipe Santana Ramires
FOR JURY EVALUATION
Mestrado Integrado em Engenharia Eletrotécnica e de Computadores
Supervisor in FEUP: Rui Penha, PhD
Supervisor in INESC: Matthew Davies, PhD
July 13, 2017
c© António Filipe Santana Ramires, 2017
Resumo
A evolução do poder de processamento dos computadores, e consequente capacidade de efetuarprocessamento de sinais digitais em tempo real, levou ao aparecimento de DAWs, tornando acriação musical acessível ao público geral. Com estas alterações, novos instrumentos e inter-faces para a criação de música eletrónica surgiram, mas continua a haver uma grande procura pornovos controladores. Os desenvolvimentos em MIR e em Machine Learning, tornaram possíveissistemas capazes de transcrever frases de bateria e de beatbox. No entanto, esses sistemas são de-senvolvidos com foco na avaliação da performance de algoritmos de transcrição e não são fáceisde usar num cenário de produção musical.
O objetivo principal deste trabalho é criar uma aplicação que permita que produtores musicaispossam usar a sua voz para criar frases de percussão quando compõem em DAWs. Um sistemafácil de utilizar, orientado para o utilizador e capaz de transcrever automaticamente vocalizaçõesde percussão, chamado LVT, é proposto. Esta aplicação foi desenvolvida usando o Max for Live esegue o método "segment-and-classify" para transcrição de bateria [1]. O LVT tem três módulos:i) um detetor de eventos, que deteta o início de uma vocalização; ii) um módulo que extrai ca-racterísticas relevantes do áudio de cada evento; e iii) uma componente de Machine Learning queimplementa o algoritmo k-nearest neighbours para classificação de vocalizações de percussão.
Devido às diferenças nas vocalizações do mesmo som de percussão de diferentes utilizadores,uma abordagem dependente do utilizador foi desenvolvida. Nesta perspetiva, o utilizador finaltem a capacidade de treinar o algoritmo com as vocalizações desejadas para cada som de bateria.Um externo para Max, que implementa o algoritmo Sequential Forward Selection, para escolheras características mais relevantes para cada utilizador é proposto, assim como um dataset anotadode vocalizações de percussão.
A avaliação do LVT feita neste trabalho tem dois objetivos. O primeiro é identificar a melhoriade performance ao ser usado um algoritmo treinado pelo utilizador final, em comparação comum algoritmo treinado por um dataset geral. O segundo objetivo é analisar se o LVT forneceao utilizador um melhor workflow para produção musical em comparação com as ferramentas jáexistentes: LDT [2] e a função do Ableton Live Convert Drums to MIDI. Os resultados mostraramque ambos os objetivos para o LVT foram alcançados.
i
ii
Abstract
The development of computers performance capacity, and consequent possibility for real-timeDigital Signal Processing (DSP) for audio, led to the appearance of Digital Audio Workstations(DAWs), making the creation of computer music available to the general public. Along with thesechanges, new instruments and interfaces for creating electronic music have surfaced. Howeverthere is still a high demand for new controllers. The developments in music information retrieval(MIR) and in machine-learning paved the way for systems capable of transcribing drum loopsand beatboxing. However, these systems are focused on evaluating the performance of transcrip-tion algorithms in offline testing scenarios and are either not easy to operate for end-users or notsufficiently reliable for use in a real music production workflow.
The primary goal of this work is to develop an application that enables music producers touse their voice to create drum patterns when composing in music DAWs. An easy-to-use anduser-oriented system capable of automatically transcribing vocalisations of percussive sounds,called LVT, is presented. This system was developed as a Max for Live device which follows the“segment-and-classify” methodology [1] for drum transcription. LVT includes three modules: i)an onset detector to segment events in time; ii) a module that extracts relevant features from theaudio content; and iii) a machine-learning component that implements the k-nearest neighbours(k-NN) algorithm for classification of vocalised drum timbres.
Due to the differences in vocalisations from distinct users for the same drum sound, a userspecific approach to vocalised transcription was developed. In this perspective, a specific end-user trains the algorithm with their own vocalisations for each drum sound before vocalising thedesired pattern. A Max external, that implements the sequential forward selection for choosingthe features most relevant for their chosen sounds, is proposed as well as a new annotated datasetof vocalised drum sounds.
The evaluation of the LVT presented in this work addresses two objectives. The first one isto identify the improvement when using a user trained algorithm instead of a dataset trained one.The second one is to assess if LVT can provide an optimised workflow for music production inAbleton Live when compared to existing drum transcription algorithms: LDT [2], and the AbletonLive Convert Drums to MIDI function. The results showed that both objectives expected for theLVT were accomplished.
iii
iv
Agradecimentos
Em primeiro lugar, gostaria de agradecer aos meus pais por todo o amor e apoio que me deram,pelo constante desejo de aumentar o meu conhecimento e por sempre terem aceitado as minhasdecisões.
À minha irmã, avós, tios e primos por sempre terem desejado o meu bem e torcerem por mim.Um agradecimento especial à Tia Lurdes, por todo o carinho e, à Mimi, pelos mimos e comidacaseira.
À Catarina, por todo o amor e carinho, por ter aturado todos os meus momentos mais difícies,dando-me força para continuar, pelo quanto me fez crescer e por fazer do meu mundo um mundobem melhor.
Aos orientadores desta dissertação, Professor Matthew Davies e Professor Rui Penha, pelasua supervisão, por terem apoiado esta ideia e me terem dado a oportunidade de trabalhar numaárea que gosto. A Matthew Davies pelo incansável apoio e interesse por esta dissertação e pelaamizade. A Rui Penha pela paixão pela música que consegue sempre transmitir.
A todos os elementos do Sound and Music Computing Group, que me apoiaram nas dificul-dades encontradas nesta tese, especialmente ao Diogo pela sua prontidão em ajudar e esclarecerqualquer dúvida.
Aos meus amigos que sempre lá estiveram para mim e que fortaleceram a minha paixão pelamúsica: Chico, Bicá, Brás, Craveiro, Gonçalo, Martins, Costa, Alex, Cavaleiro e Sérgio.
A todos os que participaram na recolha do dataset e à Rádio Universidade de Coimbra.A todos os outros colegas e amigos que me apoiaram ao longo da vida, especialmente na
Universidade de Coimbra.
António Ramires
v
vi
“Qualquer criador quando estimula outro para fazerem coisas deve sentir-se contente.”
António Pinho Vargas in "À Procura da Perfeita Repetição"
vii
viii
Contents
Abstract i
Acknowledgements v
Abbreviations xv
1 Introduction 11.1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Dissertation Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 Publication Resulting from this Dissertation . . . . . . . . . . . . . . . . . . . . 2
2 Background and State of the Art 32.1 Vocalised Percussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Electronic Music Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2.1 Electronic Music Composition Tools . . . . . . . . . . . . . . . . . . . . 42.3 Music Information Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.4 Drum Transcription . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5 Vocalised Percussion Transcription . . . . . . . . . . . . . . . . . . . . . . . . . 92.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3 Problem Characterization 133.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.2 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4 Methodology 154.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4.1.1 Onset Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.1.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.1.3 Feature Selection and Machine Learning Algorithm . . . . . . . . . . . . 18
4.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.2.1 Onset Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.2.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.2.3 Feature Selection and Machine Learning Algorithm . . . . . . . . . . . . 21
4.3 User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.3.1 LVT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.3.2 LVT Receiver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
ix
x CONTENTS
5 Data Preparation 275.1 Dataset Recording . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.2 Dataset Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6 Evaluation 316.1 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
7 Conclusions 417.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427.3 Perspectives on the Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
A seqfeatsel C Code 45A.1 seqfeatsel Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46A.2 Flowchart of the seqfeatsel external . . . . . . . . . . . . . . . . . . . . . . . . 55
References 57
List of Figures
3.1 Different kick drums waveforms overlaid with spectogram. From left to right:Drum kit, beatboxer and vocalised kick drum. . . . . . . . . . . . . . . . . . . . 13
4.1 Flowchart summarising the system . . . . . . . . . . . . . . . . . . . . . . . . . 164.2 Main part of the Max patch responsible for the operation of the system and its
components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.3 Inside the pfft∼ patch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.4 User interface of the LVT device . . . . . . . . . . . . . . . . . . . . . . . . . . 254.5 User interface of LVT receiver device . . . . . . . . . . . . . . . . . . . . . . . 26
5.1 Pattern participants were asked to reproduce . . . . . . . . . . . . . . . . . . . . 275.2 Organization of the dataset files in an Ableton Live project . . . . . . . . . . . . 295.3 Example the audio annotation in Sonic Visualizer . . . . . . . . . . . . . . . . . 305.4 Example of how participants vocalised the pattern . . . . . . . . . . . . . . . . . 305.5 Two different vocalisations of kick drum . . . . . . . . . . . . . . . . . . . . . . 305.6 Two different vocalisations of a snare drum . . . . . . . . . . . . . . . . . . . . 30
6.1 Ableton project for the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 326.2 Desired Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.3 How the number of operations was calculated. 1) Delete the extra events; 2) Cor-
rect the events that can be corrected; 3) Add the missing events. . . . . . . . . . 336.4 Effect of changing the window size per vocalised drum sounds and across micro-
phones. All LDT scores are shown in red, Ableton Live (ABL) in green and LVTin blue. The solid lines indicate the laptop microphone, the dotted lines the AKGmicrophone, and the dashed lines the iPad microphone. . . . . . . . . . . . . . . 35
6.5 Transcription of the first user vocalisations using the LVT system trained by thesecond user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.6 Transcription of the second user vocalisations using the LVT system trained by thefirst user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.7 Effect of choosing a wrong feature for a user. a) 2nd user; b)1st user with featurefor 2nd user; c)1st user. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.8 Example of an LVT transcription . . . . . . . . . . . . . . . . . . . . . . . . . . 376.9 Example of an Ableton Live Convert Drums to MIDI transcription . . . . . . . . 376.10 Example of a LDT transcription . . . . . . . . . . . . . . . . . . . . . . . . . . 37
A.1 Flowchart of the seqfeatsel external . . . . . . . . . . . . . . . . . . . . . . . . 56
xi
xii LIST OF FIGURES
List of Tables
2.1 Summary of the different vocalised percussion approaches . . . . . . . . . . . . 12
5.1 Number of individual hits contained in the recordings . . . . . . . . . . . . . . . 28
6.1 F-measure results for the PC microphone . . . . . . . . . . . . . . . . . . . . . 346.2 F-measure results for the AKG microphone . . . . . . . . . . . . . . . . . . . . 346.3 F-measure results for the iPad microphone . . . . . . . . . . . . . . . . . . . . . 346.4 Number of Operations for the PC microphone . . . . . . . . . . . . . . . . . . . 386.5 Number of Operations for the AKG c4000b microphone . . . . . . . . . . . . . 386.6 Number of Operations for the iPad microphone . . . . . . . . . . . . . . . . . . 38
xiii
xiv LIST OF TABLES
Abbreviations
DAW - Digital Audio Workstationk-NN - k-Nearest NeighboursACE - Autonomous Classification EngineRMS - Root Mean SquareFFT - Fast Fourier TransformANN - Artifical Neural NetworkMFCC - Mel frequency Cepstral CoefficientsBFCC - Bark frequency Cepstral CoefficientsMIR - Music Information RetrievalSVM - Support Vector MachinesGMM - Gaussian Mixture ModelsHMM - Hidden Markov Model IF - Instance Filtering SNR - Signal to Noise RatioNMF - Non-negative Matrix FactorisationDSP - Digital Signal ProcessingMIDI - Musical Instrument Digital Interface
xv
Chapter 1
Introduction
1.1 Context
Music culture has changed a lot in the past years. New music genres were created, new possibili-
ties of production were discovered and new instruments were tested. In this context, in particular
with the emergence of drum machines, different ways of expressing percussive patterns have sur-
faced. The most common interfaces either use pads or a sequencer in order to acquire rhythmic
representations. These tools fail to fulfil their task if the user is not able to reproduce the desired
pattern through finger-drumming or by sequencing it. The human voice is an easy and cheap way
to express a drum pattern. With the development of computers, software Digital Audio Work-
stations began to emerge, and Ableton Live, through the use of the "Convert Drums" to MIDI
function, is able to transcribe drum recordings to a MIDI pattern. The transcription presented by
this function is not accurate if the voiced input does not realistically mimic expected drum sounds,
such as the ones from a drum machine or a drum kit. Therefore this project aims to design an
interface to express drum patterns through the use of human voice.
1.2 Goals
The objectives defined for this project are the following:
• Compile a dataset of vocalised drum patterns to be available online.
• Conceive methods for automatic transcription of vocalised percussion.
• Research techniques for the incorporation of user-input.
• Create a Max for Live device to transcribe vocalised percussion.
• Evaluate the device and compare it to the existent solutions.
1
2 Introduction
1.3 Motivation
With the changes that occurred in music culture, music production and the way musicians work
with their instruments have also changed [3]. The ability to invent and reinvent new ways to pro-
duce music is nowadays a key to progress. Consequently, new proposals, such as designing new
techniques for the composition of music, are necessary. Within the genre of electronic music, the
sequencing of drum patterns plays a critical role. The voice is an important and powerful instru-
ment of rhythm production [4] and it can be used to express a drum pattern. In order to leverage
this concept within a computational system, we create a tool that can help users (both expert mu-
sicians and amateur enthusiasts) input the rhythm patterns they have in mind to a sequencer, via
automatic transcription of vocalised percussion. Our proposed tool is beneficial both from the per-
spective of workflow optimisation (by providing accurate real-time transcriptions) and as a means
to encourage users to engage with technology in the pursuit of creative activities.
1.4 Dissertation Structure
Besides the Introduction, this dissertation contains seven Chapters. In Chapter 2, the state of the
art is described and the evolution of the work in this area is presented. In Chapter 3, the obstacles
we were met with in this project as well as the proposed solution to overcome them are detailed.
In Chapter 4, a description of the theoretical and practical implementation of the LVT system
is given. Chapter 5 describes the procedure used to collect, organise and annotate the dataset. In
Chapter 6, the test methodology and the results for the evaluation of the state of the art systems
and of LVT are presented. Finally, in Chapter 7, the results and contributions of this work are
summarised and future work is proposed.
1.5 Publication Resulting from this Dissertation
This dissertation led to the presentation of the following paper:
• A. Ramires, M. Davies and R. Penha, “Automatic Transcription of Vocalised Percussion” in
DCE17 - 2nd Doctoral Congress in Engineering, 2017.
Chapter 2
Background and State of the Art
2.1 Vocalised Percussion
Vocalised percussion is one of the most intuitive ways for humans to express a rhythm. This
universal language uses phonemes with no meaning to mimic instruments and has been used in
many different cultures, throughout history, either to represent and teach percussive patterns or as
a percussive instrument itself [4] [5].
Firstly, in Australia, the ‘didjeridu talk’ or ‘tongue talk’ is used to memorise and guide a yidaki
(didjeridu) performance and the onomatopoeias used vary between different communities. Both
the conga players from Cuba and the Ewe people from Ghana use vocalised percussion sounds to
speak their riffs. In Asia, the bols and Konnakol from India comprise sound symbols to represent
different tabla hits. In the case of Europe, vocalisation is also used as an instrument itself. The
puirt a’ bhèil from Scotland and Ireland is used as an instrument substitute when the fiddles and
pipes are not available [5].
In the United States, in addition to early examples of jazz skat singing, we have beatboxing,
a form of vocal percussion originated in 1980’s hip-hop culture, where musicians use their lips,
cheeks, and throat to create different beats. This term originally referred to the vocal reproduction
of 80’s drum-machines, also known as beatboxes, that were not affordable by the vast majority of
people from this culture. With the evolution of beatboxing and with the use of microphones, the
range of expressions that beatboxers use is not simply restricted to drum sounds. By using inhaled
and exhaled sounds, different vocal modes such as head voice, growl or falsetto and trills, rolls
and buzzes, the beatboxer can create sounds such as a vocal scratch or a "synth kick" [6] [5] [4]
[7].
2.2 Electronic Music Production
With the increase in computer’s performance capacity, it became possible to do real-time Digi-
tal Signal Processing (DSP) on their personal systems and the emphasis in music-making tools
has gone from hardware to software, and the general public can now make music on their home
3
4 Background and State of the Art
computers. While computer music has been performed in academic research and composition
communities for many years, the availability of accessible software music tools has given rise to
a computer music culture outside these circles. Many exciting kinds of music are being made by
non-academic artists and producers in home studios all over the world [3].
Electronically produced music is part of popular culture. Musical ideas that were once con-
sidered far out, such as the use of environmental sounds, ambient music, turntable music, digital
sampling, computer music, the electronic modification of acoustic sounds, and music made from
fragments of speech, have now been incorporated into popular music. Genres including new age,
rap, hip-hop, electronic music, techno and jazz have been influenced with production values and
techniques that originated with classic electronic music [8].
2.2.1 Electronic Music Composition Tools
Various inventions have been devised to assist musicians in performing, arranging, recording and
composing music. A historically early method of recording music which is still in use today is the
player piano. Holes, corresponding to particular notes, are punched in paper which is rotated as
the player piano is played [9]. Newer tools used in both analogue and software electronic music
production comprise sounds generators, effects processors and mixers [10].
Any technology that can transduce human gesture or movement into electrical signal is avail-
able to be used as an electronic music composition tool. The commonly used technologies include
infrared, ultrasonic, hall effect, electromagnetic and video. With the development of MIDI, com-
puter hardware such as keyboards, switches, pushbuttons, sliders, joysticks or drum pads can also
be used as a means to input patterns and melodies in the computer. Many musicians have built
their own input devices as prototypes by using microphones, accelerometers and other types of
sensors combined to electronic circuitry. But all this hardware only works if it is connected to the
computer and managed by some software - the performance software [11].
2.3 Music Information Retrieval
Music Information Retrieval (MIR) is concerned with the extraction and inference of meaningful
features from music, indexing of music using these features and search and retrieval schemes, as
defined by [12] [13].
During the 2000s, with the development of computers and the corresponding increase in com-
puter power, MIR research shifted its focus from analysing symbolic representation of music
pieces to the use of signal processing techniques directly to the music audio signals [13].
According to Schedl et al., in [13], MIR comprises several investigation subfields. The most
typical ones are the following:
2.3 Music Information Retrieval 5
• Feature Extraction: This first group of topics is related to the extraction of relevant features
from music content. It includes several tasks such as timbre description; music transcrip-
tion and melody extraction; onset detection, beat tracking and tempo estimation; tonality
estimation and structural analysis, segmentation and summarisation.
• Similarity: This subsystem is the core of many applications such as music retrieval and mu-
sic recommendation systems. This comprises tasks such as similarity measurement, identi-
fication of cover songs and query by humming.
• Classification: This group uses the information retrieved by the previous subfields in order
to classify music. Emotion and mood recognition; music genre classification; instrument
classification; composer, singer and artist identification and auto-tagging are common areas
of research.
• Applications: This final subfield comprises the development of application that use MIR
tools. These can vary from audio fingerprinting to playlist generation and music visualisa-
tion.
Feature extraction is essential to this work and its tasks will, therefore, be described more
thoroughly.
Automatic music transcription, according to [14] "is the process of converting an acoustic
musical signal into some form of musical notation". This area is one of the most intensively re-
searched in MIR and is often considered the core technology to improve any MIR system. While
most publications deal with pitched instruments, rhythm extraction is also a major focus. A com-
plete music transcription system comprises various sub-systems such as multi-pitch detection, on-
set detection, instrument recognition and rhythm extraction. A large subset of current approaches
for the transcription of harmonic sounds employs spectogram factorisation techniques, such as
NMF and probabilistic latent component analysis [15] [16].
Beat tracking is defined in [17] as deriving from a music audio signal a sequence of beat
instants that might correspond to when a human listener would tap his or her foot. This task is
related both to note onset detection, which consists in identifying the start points of musical events
in an audio signal, and tempo induction, which resides in finding the underlying rate of a piece of
music [18]. Despite the differences between these two tasks, their investigation has always been
closely connected. Research on this topic started in the 1970s and an overview on its evolution
can be found in [19].
Research on structural analysis or self-similarity analysis mainly consists in detecting signal
changes and repetitions, within the same musical piece. This analysis is based on the computation
of the self-similarity matrix, proposed by Foote in [20]. An important application of this research
is music summarisation, as songs may be represented by their most frequently repeated segments
[13].
6 Background and State of the Art
2.4 Drum Transcription
Drum transcription is essential in automatic music transcription as, in several music genres, the
drum track possesses information about tempo, rhythm, style and possibly the structure of the song
[15]. Various problems can arise when dealing with this task. These are related to the diversity of
the drum sounds to be labelled, the difference in loudness in different loops and the possibility of
overlapping sounds.
Most drum transcription methods can be separated in three different groups, as proposed in [1]
and [21]:
• Segment and Classify: This first approach segments different drum events and, based on
the features extracted, classifies them using machine learning techniques, such as support
vector machine (SVM) or gaussian mixture models (GMM). This proved successful in solo
drum recordings, but, in polyphonic music, its application is more challenging, as most of
the features used for classification are sensitive to the presence of background music.
• Separate and Detect: The input signal is split in its various components via source separa-
tion. The different streams then go through an onset detector, such as an energy threshold
based one, in order to find the instances of each signal. To achieve source separation, a
time-frequency transform is normally used. This decomposition is traditionally achieved
with independent subspace analysis (ISA) or non-negative matrix factorisation (NMF).
• Match and Adapt: These methods search for the occurrence of a temporal or time-frequency
templates within the music signal and browse a database to find the most similar pattern to
the queried one.
The "segment and classify" and "separate and detect" methods are the most relevant ones
for this work, as they do not need previously created templates to match the query. In order to
power creativity, the user should be able to input any sequence of their own design, and not only
previously constructed ones.
Gillet et al. [22], study the performance of hidden Markov models (HMM) and SVMs on the
transcription of drum loops in a "segment and classify" method. Instead of focusing the identifi-
cation only on sounds taken in isolation, the dataset used consists of pre-recorded drum patterns,
such as those found in commercial sample CDs, where an event can contain more than one drum
hit. In order to split the loop into the corresponding events, an onset detection algorithm based
on sub-band decomposition was used. A k-NN classifier was used to find the most appropriate
group of features to be used in the event classification sub-system. The selected features were:
(1) the mean of 13 Mel frequency Cepstral Coefficients (MFCC); (2) the spectral centroid; (3) the
spectral width; (4) the spectral asymmetry; (5) the spectral flatness; (6) 6 band-wise frequency
content parameters, that correspond to the log-energy in six pre-defined bands. The classification
was done both with an HMM and an SVM. The first class of models was used as it proved efficient
when short term time dependencies exist. This is the case if the sound produced by a drum con-
tinues to resonate when the following stroke happens. Both classes were tested with two different
2.4 Drum Transcription 7
approaches. The first one using only one 2n-ary classifier, in which each possible combination
of strokes is represented as a separate class. The second one uses n binary classifiers, one per
instrument. A third experiment was conducted using a drum kit dependent approach, where four
different HMM classifiers were used, one for each kind of drum kit (Electro, Light, Heavy or Hip-
Hop). The results obtained show that the SVM surpasses the HMMs in all approaches, acquiring
65.1% accuracy using only one classifier, and 64.8% using n binary classifiers. The highest accu-
racy for the HMM classifier occurred when the drum kit dependent approach was used, attaining a
precision of 62.5%, while only 59.1% was achieved in the model trained on all data. The authors
state this probably was due to the high variability of the data set, which the HMM approach could
not handle.
Gillet et al. [23], in order to remove the non-percussive parts of polyphonic music, use a
band-wise harmonic/noise decomposition. This algorithm is only able to identify two classes:
kick and snare. The aim of the first stage of this system is to obtain the stochastic part of the input
signal. Percussive sounds have a strong stochastic component, contrary to the pitched instruments.
In order to achieve this, the input signal is decomposed into eight non-overlapping sub-bands, by
passing it through an octave-band filter bank, since, in this way, the computational cost of the noise
subspace projection is greatly reduced. The second stage is to project the noise subspace. This
is accomplished by using the Exponentially Damped Sinusoidal model. After this, the signals
still contain attacks and transients from pitched instruments, therefore, in the next stage, where
the resulting signal goes to an onset detection algorithm, non-percussive events are also detected.
This is handled by adding another class for these sounds. The onset detection is done by half-
wave rectifying and low-pass filtering the sub-band noise signals, and then finding the peaks of
their derivative. The features extracted from each segment were the energy in the first 6 sub-bands
and the average of the first 12 MFCCs, without c0. The classification used is different from the
standard "segment and classify" approach as some onsets do not contain drum events and have
to be discarded. Two SVM classifiers were used, one for the kick and one for the snare. The
probabilistic output of the SVM classifier was used as a likelihood measure, in order to retrain the
system with the most probable events. The best F-measure achieved was 89.2% for a mix where
the drum was 6dB louder than the rest, and 84.0% for a balanced mix.
Similarly to Gillet et al. in [23], Tanghe et al., in [24], present a strategy to segment and
classify drum events, in real time, in polyphonic music, using an SVM. The algorithm operates in
a streaming way, which allows the processing of big audio files and never ending audio streams.
The onset detection consists of several sub-systems, in order to only detect local maxima, and
detects both drum and non-drum events. Firstly, the audio signal is fed into a short term Fourier
transform and then input to a Mel filterbank. The weighted sum of the differences between the
current amplitude levels and that of the recent past is calculated. Here, the more recent the value
is, the more important it is. By dividing the envelope follower of the output of the Mel filterbank
by the result of the weighted sum, the relative differences in each frequency band are calculated.
The output of this sub-system goes to a peak detector and, if this peak is higher than a selected
threshold, it is sent to a heuristic grouping peak detector that outputs “true” if a local maximum
8 Background and State of the Art
is reached and “false” if not. A new peak can only be detected if the calculated sum decreases
after the previous peak is identified. A module was created to extract features from the stream of
audio samples. The descriptors obtained were the overall RMS, the RMS in 3 frequency bands, the
RMS per band relative to the overall RMS, the RMS per band relative to RMS of other bands, the
zero-crossing rate, the crest factor, the temporal centroid, the spectral centroid, kurtosis, skewness,
rolloff and flatness and the MFCC and ∆ MFCC. The classification is then executed by an SVM,
trained with annotated audio files. The highest average classification F-measure achieved was
61.1%.
Miron et al. [2] introduce a drum transcription algorithm capable of handling real-time audio.
In order to detect events, first, a high frequency onset detection is used, since this was reported
to be the best for percussive onsets. As this stage can detect false positives, an instance filtering
(IF) method using sub-band onset detection is used. This method uses the complex domain onset
detector in three different frequency bands, one for each class (kick, snare and hi-hat). Since
different drum strokes can occur at the same time, features for each frequency band are computed
separately, so that the noise influence is reduced. The obtained features are computed in the decay
part of each sound and, in order to give less importance to silent frames, weighted with the RMS
value. The extracted features are energy in 23 bark bands, 23 bark frequency cepstrum coefficients,
spectral centroid and spectral rolloff. The machine learning part consists of three k-NN classifiers,
each adapted to the corresponding class. This system is implemented in PD-extended, Max MSP
and in Max for Live. The F-measure result obtained by the event detection sub-system was 93%,
and by the complete system was of 81% for the validation dataset. The use of the IF stage along
with the k-NN classifier led to an increase in the performance and precision in all classes.
A "separate and detect" method for drum transcription was presented by Roebel et al. [25].
The separation of the three sources is done using a non-negative matrix deconvolution, in which
the update rules are obtained from the Itakuro Saito divergence. The detection of drum events uses
three criteria. The first one comprises an activation based test. The second threshold is used in
order to establish a minimum SNR for detected events. The prominence of the target class in the
power spectrum is weighted and compared to a third threshold. If all test are passed, the event is
retained. The use of these three conditions show a better overall performance in the algorithm.
The non-negative matrix factorisation approach can also be used in a "match and adapt"
method, as proposed by Wu et al. [15]. A template adaptive drum transcription algorithm that
uses partially fixed non-negative matrix factorisation is presented. This algorithm uses two dictio-
naries: one previously defined with drum templates and one trained with melodic content, in the
standard NMF manner. Two methods are then tested for the template adaption. For three classes
of drums, the system is able to achieve average F-measures of 77.9% and 72.2% in monophonic
and polyphonic music respectively.
2.5 Vocalised Percussion Transcription 9
2.5 Vocalised Percussion Transcription
The transcription of vocalised percussion deals with automatic identification of vocalisations of
percussive sounds. This area has many applications such as live music transcription, human-
machine musical interaction or even identifying drum loops within a database [26].
Most systems for the transcription of vocalised percussion follow the "segment and classify"
approach and integrate three different parts: a component that separates the different events, a
component that generates descriptors for each event and a machine learning component that as-
signs the different events to the corresponding class [27]. Therefore, vocalised percussion is a
monophonic transcription problem.
Amaury Hazan et al. [27] created a tool for transcribing voice percussive rhythms that aims to
reduce the gap between the user and the device used for acquiring rhythmic representation. This
work focus in transcribing not only standard vocalised drum sounds but also a whole range of
acoustic oral rhythms. In order to separate the different events, an energy based algorithm is used.
This decomposes the input stream into several frames, computes their energy and compares it to
a threshold that is user-defined. In order to obtain descriptors, each event is split into attack and
decay, by finding the maximum of the event’s sound envelope. The features to be extracted were
divided between temporal and spectral. From the decay, both temporal and spectral descriptors
were obtained, whereas in the attack only temporal features were analysed. The spectral descrip-
tors used were obtained through the use of an FFT. These descriptors are: spectral energy, spectral
centroid, flatness, kurtosis and the first five MFCC. In the case of temporal features, the descriptors
used were the duration, the log-duration, the energy, the zero-crossing rate and the temporal cen-
troid. Two different machine learning components were used. The first one was the tree induction
algorithm method C4.5, with and without two optimisations: boosting and bagging. The second
algorithm used was the k-NN. The attained results favoured the utilisation of the C4.5 algorithm
with bagging with 90% accuracy for a test set with recordings from unseen performers, compared
to 87% accuracy for the C4.5 algorithm with boosting, and 79% both for the C4.5 algorithm alone
and for the k-NN.
In the report "Automatic Transcription of Beatboxing" by Christensen et al. [28], the group
developed a MATLAB application that identifies three beatboxing sounds, the kick, the snare and
the hi-hat. In order to segment different events, they use an energy based algorithm similar to
the one in the paper by Amaury Hazan et al. [27]. The classification was done using the k-NN
classifier, previously trained with a beatbox dataset recorded by them. A choice was made to
only test one feature at a time. These were energy, zero crossing rate, the first 20 MFCCs and
spectral centroid, spread and flux. Later, they analysed which k values showed better results for
each feature used. The best performing feature was the MFCC feature vector, with k = 7 or 8. An
accuracy of 98.9% was achieved when using these parameters.
In their paper [4], besides presenting a data collection comprising recordings from both beat-
boxers and non-beatboxers, Sinyor et al. studied the efficiency of the Autonomous Classification
Engine in identifying vocalised percussion sounds. This engine optimises the set of features to
10 Background and State of the Art
be used by the machine learning component [29]. A second experiment was performed using a
genetic algorithm for selecting features that proved to be superior compared to the 1-NN ACE
experiment. The segmentation of the input was done manually and the descriptors used were both
the average and the standard deviation of compactness, of spectral rolloff, of RMS derivative and
of zero crossing overall, the standard deviation of the overal RMS and of the frequency corre-
sponding to the highest peak of the FFT, and the average of both the the zero crossing derivative
and the strongest frequency of the spectral centroid. The classifier that proved to work best with
ACE was AdaBoost with C4.5 decision trees as base learners, which obtained a 98.15% accuracy
when using 3 classes of sounds, and 95.55% when using 5 classes.
Kapur et al. [7] introduce two different systems. The first one receives a beatboxing loop
and identifies the corresponding drum loop within a bank. The second one transcribes the same
input to the corresponding drum sounds (kick, snare and hi-hat). Despite the first system being
an interesting utilisation for beatbox transcription, we will focus on the transcription application.
The segmentation of the input is also done by splitting the audio when its volume is higher than
a definable value. The classification algorithm used was a backpropagation Artificial Neural Net-
work with a single dimensional feature that was the number of Zero Crossings. This method was
used since it was the one to achieve best results in a real time implementation, having obtained
an accuracy of 97.3%. In this method each of the drum sounds should be recorded 4 times before
the transcription, in order to train the ANN. The user may then record a beatboxing loop, that will
be processed and fed to a sampler, where the user can select the desired real drum sounds. The
transformed query can the be saved as a new audio file.
In [26], Stowell et al. study the effect of delaying the classification in a real time transcription
system and present a new annotated beatbox dataset. They choose to have 3 classes that are kick,
snare and hi-hat. The onset detection was done manually, in order to factor out the influence of
this component in the system. Several different features were obtained through SuperCollider
3.3, and then analysed to see the corresponding effect in the accuracy of the transcription by the
naive Bayes classifier, with the Kullback-Leibler divergence. The best value of time frames to
be analysed for each feature was also tested. The features that proved to be more appropriate for
beatbox transcription were the 25th- and 50th-percentile and the spectral centroid and flux. The
delay that performed best in most tests was 23ms. The obtained accuracy with these parameters
was 88.4 % for the kick, 81.6% for the hi-hat and 53.1% for the snare. A perceptual experiment
was also conducted in order to evaluate the tolerable latency in the decision-making component.
For common drum sounds, the best maximum delay which preserved an excellent or good audio
quality varied from 12ms to 35ms.
Nakano et al. [30] present a "match and adapt" method to retrieve drum patterns from a
database by voice percussion recognition. By using the Viterbi algorithm and only two sound
classes (kick and snare), a recognition rate of the desired pattern of 93% is attained. Gillet et
al. [31] present a system, with the same function as the previous one, that uses a segment and
classify approach. The input is manually segmented and the features used are 13 MFCCs and 13
∆ MFCCs. The transcription of the query is done by using the Bakis (left-right) HMM model.
2.6 Summary 11
Hipke et al. [32] present a transcription system which uses an identifier that is trained by
the end user. This system is named BeatBox and enables end-user creation of custom beatbox
recognisers, represented in the GUI by different pads. Each pad also shows the reliability of
the differentiation for each vocalisation. The onset detection is threshold based and the features
computed are the spectral centroid and RMS. The classification algorithm used was k-NN, which
k value is automatically selected by the system.
The DAW Ableton Live has a "Convert Drums to New MIDI Track" function that "extracts
the rhythms from unpitched, percussive audio and places them into a clip on a new MIDI track"
and should be able to work "with your own recordings such as beatboxing" [33]. In the context of
this work, this is the most similar system to the one we propose, as Ableton Live is a DAW widely
used by both expert musicians and amateur enthusiasts. This feature works satisfactorily when
transcribing drums and beatboxing recordings that imitate drum sets. By testing it with simple
onomatopoeia such as boom, pam, ta, pa or tss the transcription did not work as intended. In
recordings with a cheap microphone, the noise was identified as a hi-hat. Moreover, some of the
vocalisations of snares or kicks were identified as a snare and kick at the same time.
Table 2.1 resumes the previous articles in terms of accuracy, number of classes used, type of
segmentation, the descriptors used and the machine learning algorithm adopted.
2.6 Summary
As was described in this chapter, there are many articles focused on the complications regarding
the transcription of percussive sounds. The presented systems are mostly focused on evaluating
the performance of transcription algorithms and not on the possible applications to end-users.
Despite the satisfactory results, the transcription of percussion presents highly diverse prob-
lems in need of techniques specifically adapted to the main target scenario, whether the input of
the system consists in vocalised percussion, beatboxing, isolated or mixed polyphonic recordings.
12 Background and State of the Art
Table 2.1: Summary of the different vocalised percussion approaches
Author NumClass
Segmentation Descriptors Used MachineLearning
Accuracyachieved
Hazan [27] 4 EnergyThreshold
Spectral energy, spectral cen-troid, flatness, kurtosis, the firstfive MFCC,duration, the log-duration, the energy, the zero-crossing rate and the temporalcentroid
C4.5w/Bagging
90%
Christensen[28]
3 EnergyThreshold
20 MFCC k-NN 98.9%
Sinyor [4] 5 Manual Average and standard deviationof compactness, spectral rolloff,RMS derivative and zero cross-ing overall; standard deviationof the overall RMS and of themaximum frequency and aver-age of the zero crossing deriva-tive and of the strongest fre-quency of the spectral centroid.
C4.5 w/-Boosting
95.55%
Sinyor [4] 3 Manual Average and standard deviationof compactness, spectral rolloff,RMS,derivative and zero cross-ing overall; standard deviationof the overall RMS and of themaximum frequency and aver-age of the zero crossing deriva-tive and of the strongest fre-quency of the spectral centroid.
C4.5 w/-Boosting
98.15%
Kapur [7] 3 EnergyThreshold
Number of zero crossings ANN 97.3%
Stowell [26] 3 Manual The first 8 MFCCs, the spectralcentroid, spread, flatness, flux,slope,crest, crest in subbandsand distribution percentiles, thehigh-frequency content and thezero-crossing rate.
naiveBayes
Kick 88.4%Snare 81.6%Hi hat 53.1%
Chapter 3
Problem Characterization
3.1 Problem Definition
The problem consists on the creation of a tool that assists producers, either trained in beatboxing
or not, to create patterns with the use of their own voice. This application should receive as
input either a recording or a stream of audio that contains vocalised percussive sounds and output
the corresponding transcription in a ready-to-use MIDI file. Furthermore, the system should be
adapted to vocalised input constraints, such as the inability of producing two sounds at the same
time.
The applications stated in the previous chapter do not suffice if a user-specific vocalised per-
cussion transcription system, aimed at computer musicians, is desired. These tools are aimed at
testing the behaviour of transcription systems, they export results instead of patterns and their
interfaces are not easy to use. Moreover, most of them are aimed at either beatboxing or drum
transcription and are tuned to receive only a selection of sounds.
Figure 3.1: Different kick drums waveforms overlaid with spectogram. From left to right: Drumkit, beatboxer and vocalised kick drum.
A substantial difference exists between the sound produced by a drum kit, by a beatboxer and
13
14 Problem Characterization
by a common user not trained in beatboxing as can be seen informally in Figure 3.1. In addition,
different people vocalise drum sounds in different manners.
Therefore, for this tool to function as required, it has to be adapted to each user and to the
characteristics of the human voice.
3.2 Proposed Solution
In order to solve this problem, a vocalised drum transcription software, able to be trained with the
user vocalisations is proposed. The system is integrated in a Max for Live project. Max for Live
is a visual programming environment, based on Max 7, that allows users to build instruments and
effects for use within Ableton Live.
Firstly, a dataset of vocalised percussion was compiled. It was then annotated using Sonic
Visualiser1, a free application for viewing and analysing the contents of music audio files. The
recordings were saved and organised both in a compressed archive and in an Ableton Live project
file, for compatibility and to facilitate the testing of the transcription systems. These files are
hosted in the project’s web page2.
Then, the system was developed following an user-specific approach. This system follows the
"segment and classify" method previously described and integrates three elements: an onset de-
tector, a component that generates descriptors for each event, and a machine learning component.
The onset detection was done with Aubio Onset∼ [34]. The extraction of the features described
in the state of the art was done in real-time with the use of the object Zsa.mfcc∼, the library
Zsa.descriptors [35] and other Max MSP objects. The first of these tools outputs the MFCCs as
a list, the second one extracts spectral centroid, spread, slope, decrease and rolloff, a sinusoidal
model based on peaks detection and a tempered virtual fundamental [35]. The zero crossing rate
and number of zero crossings were calculated with the zerox∼ object. The machine learning com-
ponent was trained with the user’s preferred vocalisation and the features selected to show the
better results for the provided input. This was done through the use of the Sequential Forward Se-
lection method, with the most significant features selected by the accuracy obtained from testing
the training data. This metric evaluates the most adequate feature in order to achieve the maximum
separation between clusters in a machine learning algorithm. The Sequential Forward Selection
method works by selecting the most significant feature, according to a specific parameter (in this
case the accuracy obtained from testing the training data), and adding it to an initially empty set
until there are no improvements or no features remain. An user interface was created in Max for
Live, so as to facilitate the utilisation of the application.
1http://www.sonicvisualiser.org/2https://lvtsmc.wordpress.com/
Chapter 4
Methodology
In this chapter, the approach used to create an automatic system that transcribes vocalised per-
cussion is described. When developing the system, the focus was on creating a model capable of
delivering a reliable and accurate transcription and which is easily operated by music professionals
and amateur enthusiasts.
This chapter is divided in three different parts. The first one details the approach chosen for
the design of the system, the second one describes how this solution was implemented and in the
third part, the way the system should be used is described. A flowchart of the system functioning
is presented in Figure 4.1.
4.1 Approach
In this section, the operation of the system is detailed and divided into various components. Fol-
lowing [1], this device uses the segment-and-classify approach. This approach proved to be partic-
ularly successful on solo drum signals [1] but not as efficient in polyphonic sounds. In our system
it is only desired to transcribe percussive events and, therefore, this approach was chosen as it is
the most suitable option.
The first component is an onset detector, which is responsible for detecting when events occur.
When these are detected, the second module extracts the features from the relevant time frame and
outputs their value to the final stage, the machine learning and feature selection component. The
features that provide a better classification are chosen and used in a k-NN classifier.
4.1.1 Onset Detection
The onset detection algorithm used is the high frequency content (HFC). In the tests conducted
in [36], it proved to be the most effective method for identifying non-pitched percussive onsets,
detecting 96.7% of all the events and not detecting any false positive. The function was originally
proposed in [37]. This algorithm calculates the weighted mean of the amplitude for each bin. The
higher the frequency is, the more weight the bin has. It is powerful at detecting onsets that can be
modelled as bursts of white noise, such as snares and cymbal sounds.
15
16 Methodology
Figure 4.1: Flowchart summarising the system
4.1.2 Feature Extraction
The module that follows the onset detection is the feature extraction. A set of temporal and spectral
features are extracted from the incoming audio signal when an onset is detected.
The temporal features are the RMS value of the energy and the number of zero crossings. The
number of zero crossings corresponds to the number of times that a signal crosses the x-axis in a
fixed time frame, while the energy corresponds to the RMS value of the energy contained in an
audio frame. These features are calculated over a time frame of 4096 samples, so that the features
are extracted from the first 93ms of the vocalisation for audio sampled at 44.1kHz.
The spectral descriptors extracted are the following:
• Spectral Centroid: This feature corresponds to the centre of mass of a spectrum and is
connected to the perception of a sounds’ “brightness” [38]. Its value can be calculated as
follows:
µ =∑
n−1i=0 f [i]a[i]
∑n−1i=0 a[i]
(4.1)
4.1 Approach 17
where n is half of the FFT window size, i is the bin index, a[i] the corresponding amplitude
and f [i] is its frequency and is calculated as follows:
f [i] = i∗sample rate
FFT window size(4.2)
[35]
• Spectral Spread: This descriptor measures the variance of the spectral centroid:
ν =∑
n−1i=0 ( f [i]−µ)2 a[i]
∑n−1i=0 a[i]
(4.3)
[35]
• Spectral Slope: Calculates the slope of the magnitude spectrum by doing a linear regression
of it:
slope =1
∑n−1i=0 a[i]
n∑n−1i=0 f [i]a[i]−∑
n−1i=0 f [i]∑n−1
i=0 a[i]
n∑n−1i=0 f 2[i]−
(∑
n−1i=0 f [i]
)2 (4.4)
[35]
• Spectral Decrease: This feature is similar to the spectral slope as it also represents the
decreasing of the magnitude spectrum, but according to [39], is supposed to relate to human
perception:
decrease =∑
n−1i=0 a[i]−a[1]
∑n−1i=2:K a[i] (i−1)
(4.5)
[35]
• Spectral Roll-Off: Computes the frequency value so that 95% of the signal is below this
frequency. For x = rolloff point
fc[i]
∑i=0
a2[ f [i]] = xn−1
∑i=0
a2[ f [i]] (4.6)
[35]
• Spectral Skewness: Measures the asymmetry of the spectrum around its centre of mass and
was originally proposed in [39].
• Spectral Flux: Measures how quickly the energy of a signal is changing and is calculated
as the Euclidean distance between two normalised spectra. [40]
18 Methodology
• Spectral Kurtosis: Is similar to skewness but, instead of measuring the asymmetry of the
spectrum, it measures the flatness around its centre of mass.
• Spectral Flatness: Provides a measure of how similar to white noise a sound is and is mea-
sured in four frequency bands (250-500Hz, 500-1000Hz, 1000-2000Hz and 2000-4000Hz)
[41].
• MFCC: These are the coefficients that form a mel-frequency cepstrum and are commonly
used in speech recognition. They represent the spectrum based on its perception and is
considered in [42] to be the “best available approximation of human ear”.
• BFCC: This method is similar to the MFCC, but uses the Bark frequency filter bank instead
of the Mel filters [42].
The extracted features are normalised and, if the output is a frequency value, the scale is
changed from exponential to linear so that the lower frequencies are as important as high frequen-
cies.
4.1.3 Feature Selection and Machine Learning Algorithm
The features extracted in the previous module are given to the feature selection object that is
connected to a machine learning algorithm.
Our system is meant to be user-specific. The classification is adapted to each user and not to
a general dataset. Therefore, each user should train the algorithm with their own vocalisations,
in order to obtain a higher accuracy of prediction. The features that better differentiate each
drum sound vocalisation for each user should be chosen automatically without user interference.
To achieve this, we implemented a feature selection method which is the Sequential Forward
Selection, to be used along with the k-NN machine learning algorithm.
The SFS method was first proposed in [43] and is a bottom-up feature selection algorithm,
which means that it starts with an empty set of features. The feature that provides a best measure-
ment is initially added to this set. Additional features are added sequentially to this set until the
stopping condition is met. This condition normally is a threshold in the performance of the system
or a number of features selected.
The user should first train the system with vocalisations of each drum sound. Therefore the
SFS can use the number of correct k-NN guesses from the training set as a measure of efficiency
for each feature. This was the approach chosen in order to select the features that work better for
each user’s vocalisations. Whenever there is no improvement with the addition of features, the
algorithm stops.
The machine learning algorithm used is the k-Nearest Neighbour. This is a simple method
based on the measurement of the distances between the training data and the input sample. The
Euclidean distance is a common measure of distance between points and can be calculated as√∑
ni=1(xi − yi)2, where x and y are the two points, i the index of the axis and n the total number
4.2 Implementation 19
of axis in the Euclidean n-space. The input sample is classified as the class that has k samples less
distant to it [44].
4.2 Implementation
The system was implemented as a Max for Live device, in order for it to be easy to install and to
work with for Ableton Live users. Max for Live is a toolkit that allows to build devices to be used
in Ableton Live, using the visual programming language Max.
The implementation of the most relevant part of the back end system is shown in Figure 4.2.
Figure 4.2: Main part of the Max patch responsible for the operation of the system and its compo-nents
4.2.1 Onset Detection
Before the audio input that comes from Ableton Live is given to the onset detector, the left and
right channel are summed to mono. This way, there is only one signal chain in the system instead
of two audio channels to be analysed separately.
In Max, the HFC algorithm for onset detection can be implemented using the external object
aubioOnset∼. Aubio [45] is a free and open source C library designed for the extraction of anno-
tations from audio signals. Some of the functions provided by this library were wrapped as Pure
Data externals and, later, the onset detection function was adapted for Max MSP as an external
20 Methodology
by Marius Miron. This object receives one audio signal and, when a peak is detected, outputs a
“bang”.
The aubioOnset∼ is initialised as “aubioOnset∼ hfc 512 128 -70 0.7”. Therefore, the param-
eters used in our system are the following:
• Onset Detection Function: HFC. As previously shown, high frequency content is the most
appropriate method to use when detecting non pitched percussive sounds, which is the case
of the vocalised percussion sounds.
• Threshold: 0.7. This parameter controls the threshold value for the onset peak picking
and the values should be between 0.001 and 0.9 [34]. Different values were tested for
the threshold of the onset detection algorithm. Different audio clips from the dataset were
analysed and 0.7 provided a good balance between detecting most of the vocalisations while
not detecting many false positives.
• Silence Threshold: -70dB. This option corresponds to the volume under which the onsets
will not be detected.
• Buffer Size: 512 samples. This value deals with the number of samples that are present in
the buffer to be analysed. It also corresponds to the window of the spectral and temporal
computations [34]. The bigger this buffer is, the bigger the frequency resolution will be and
the longer it will take to detect an onset. A buffer size of 512 samples provides an accurate
detection of onsets and only corresponds to a delay of approximately 11.6ms if the sample
rate is 44.1kHz.
• Hop Size: 128 samples. This parameter corresponds to the number of samples between two
consecutive analysis frames [34]. The selected value provided a good temporal resolution.
4.2.2 Feature Extraction
The feature extraction is implemented by using either Max MSP native objects or the library
Zsa.Descriptors [35].
The number of zero crossings is calculated using the zerox∼ Max MSP object. This function
receives audio in its first inlet and outputs the number of times the analysed signal passed through
the X axis. The audio frame used for the analysis is the last signal vector. A signal vector is the
block that MSP uses in its operation and its size can be defined in the audio setup window. In
order to derive the energy, an envelope follower is used. The value of the envelope is stored when
an onset is detected and, unlike the rest of the features, it is not used for the classification but
for acquiring a velocity value. The sampled value is compared to a maximum and mapped to a
number between 1 and 127, that corresponds to the velocity of the given vocalisation.
In order to extract the spectral features from the audio, the Zsa.Descriptors library is used. This
library, which was developed in IRCAM by Mikhail Malt and Emmanuel Jordan, covers a large
set of descriptors and is able to work in a real-time situation. In order to increase CPU efficiency,
4.2 Implementation 21
the Zsa objects are implemented inside the same pfft∼ patch, as shown in Figure 4.3. This object
is a spectral processing manager for patchers and allows to work in Max MSP in the frequency
domain. The window size chosen for the FFT is 4096 samples, in order to provide a good fre-
quency resolution. The purpose of this system is to provide good accuracy in the identification of
the vocalisations, and is not focused on real-time applications. The overlap factor chosen for the
FFT analysis is 8, which corresponds to a hop size of 512 samples.
The number of zero crossings and all the spectral features are extracted 3584 samples after the
onset is detected. The buffer size of the onset detection module is 512 samples. An event can only
be detected in the end of the buffer. As we only want to analyse the audio frame that is after the
onset, the values of the feature extraction are only evaluated 4096 samples after the earliest onset
can happen. If the onset is at the beginning of the onset detector buffer, it will take 512 samples
to be detected and the features that correspond to this event will be extracted 512+3584 = 4096
samples after it occurred. The value of the energy is only calculated when the onset is detected, as
we want to use the maximum power to calculate the velocity, and this occurs at the beginning of
the vocalisation.
Figure 4.3: Inside the pfft∼ patch
4.2.3 Feature Selection and Machine Learning Algorithm
The k-NN algorithm was implemented by using TimbreID, a Pure Data external developed by
William Brent [46] and ported to Max by Marius Miron that implements the k-NN machine learn-
ing algorithm. It can use different metrics for the calculation of the distance but the one chosen
for this system is the Euclidean distance.
22 Methodology
As it did not exist, an external written in C that implements the Sequential Forward Selection
was created, using the Max API [47]. This external was developed to work together with Tim-
breID, the messages sent and received are adapted to this classifier. A flowchart resuming most of
the functioning and the full code can be seen in Appendix A.
When writing an external object for Max, according to Max API, there are five basic steps.
The first one corresponds to adding the ext.h and ext_obex.h header files
The object declaration follows. Here a C structure is declared that contains all the class vari-
ables. In this case, the C structure is declared as follows:
1 t y p e d e f s t r u c t _ s e q f e a t s e l2 {3 t _ o b j e c t ob ; / / t h e o b j e c t i t s e l f ( must be f i r s t )4 boo l i d e n ; / / i s i t a l r e a d y t r a i n e d o r n o t ?5 boo l f l a g ; / / has t i m b r e I D g i v e n an answer ?6 boo l debug ; / / used f o r debugg ing p u r p o s e s7 boo l f a s e ; / / i s t i m b r e I D answer a no c a r e ?8 l ong numFea tu res ; / / number o f f e a t u r e s r e c e i v e d9 l ong rowCount ; / / c o u n t s t h e rows
10 l ong u l t i m a N o t a ; / / l a s t r e c e i v e d MIDI n o t e11 l ong r e s p o s t a ; / / answer from t i m b r e I D12 l ong knn ; / / knn v a l u e13 l ong numNotas ; / / t o t a l number o f n o t e s14 s h o r t nSe l ; / / number o f s e l e c t e d f e a t u r e s15 t_member ∗ n o t a s P C l u s t e r ; / / a r r a y wi th t h e n o t e s f o r t h e c l u s t e r msg16 t _ i n s t a n c e ∗ t r a i n i n g T a b ; / / a r r a y wi th t h e r e c e i v e d f e a t u r e s17 i n t ∗ s e l C o l ; / / columns s e l e c t e d t h r o u g h s f s18 vo id ∗ a _ o u t ; / / o u t p u t t h e column t o use19 vo id ∗ b_ou t ; / / o u t p u t s t h e rows f o r t h e kNN e x t e r n a l20 } t _ s e q f e a t s e l ;
struct.c
The next step is to create an initialisation routine. When the object is loaded in Max this
routine is ran and it informs Max which methods should be run when an instance of the object is
created, destroyed or when it receives a message.
1 vo id ex t_main ( vo id ∗ r )2 {3 t _ c l a s s ∗c ;4
5 c = c l a s s _ n e w ( " s e q f e a t s e l " , ( method ) s e q f e a t s e l _ n e w , ( method ) s e q f e a t s e l _ f r e e ,( l ong ) s i z e o f ( t _ s e q f e a t s e l ) ,
6 0L , A_GIMME, 0) ;7
8 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ m e s s a g e , " l i s t " , A_GIMME, 0) ;9 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ i n 1 , " i n 1 " , A_LONG, 0) ;
10 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ i d , " i d " , 0 ) ;11 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ c l e a r , " c l e a r " , 0 ) ;12 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ d e b u g , " debug " , 0 ) ;
4.2 Implementation 23
13 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ k n n , " knn " , A_LONG, 0) ;14 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ a s s i s t , " a s s i s t " , A_CANT, 0 ) ;15
16 c l a s s _ r e g i s t e r (CLASS_BOX, c ) ;17 s e q f e a t s e l _ c l a s s = c ;18 }
initialization.c
After this, the new instance routine should be developed. In this function, all the class variables
are initialised, the memory space for the storage of the arrays is allocated and the inputs for the
object are created. The code is presented next:
1 vo id ∗ s e q f e a t s e l _ n e w ( t_symbol ∗s , l ong argc , t_a tom ∗ a rgv )2 {3 t _ s e q f e a t s e l ∗x = NULL;4 x = ( t _ s e q f e a t s e l ∗ ) o b j e c t _ a l l o c ( s e q f e a t s e l _ c l a s s ) ;5 x−>b_ou t = l i s t o u t ( x ) ;6 x−>a _ o u t = o u t l e t _ n e w ( ( t _ s e q f e a t s e l ∗ ) x , NULL) ;7 x−>n o t a s P C l u s t e r = ( t_member ∗ ) sysmem_newptr ( 0 ) ;8 x−> t r a i n i n g T a b = ( t _ i n s t a n c e ∗ ) sysmem_newptr ( 0 ) ;9 x−>s e l C o l = ( i n t ∗ ) sysmem_newptr ( 0 ) ; / / columns s e l e c t e d t h r o u g h s f s
10 i n t i n ( x , 1 ) ;11 x−>i d e n = f a l s e ;12 x−> f l a g = f a l s e ;13 x−>debug = f a l s e ;14 x−>f a s e = f a l s e ;15 x−>numFea tu res = 0 ;16 x−>rowCount = 0 ; / / c o u n t s t h e rows17 x−>u l t i m a N o t a = 0 ;18 x−> r e s p o s t a = 0 ;19 x−>knn = 1 ;20 x−>numNotas = 0 ;21 x−>nSe l = 0 ;22 r e t u r n x ;23 }
newinstance.c
Finally, the message handlers were written. These are the methods that are run when a message
arrives. As it can be seen in the initialisation routine, this object handles 7 different messages:
When a feature list is received, if the object is not trained, the number of features and the
label is stored and the number of instances is incremented. Otherwise, if the object is trained, the
received message is filtered and the object outputs the selected features.
Whenever a message containing “id” is received, this object will start the process to identify
which features provide a better performance for the module. For each feature:
• If the feature has already been selected, jump to the end.
• All instances are sent to timbreID, in order to train it with data.
24 Methodology
• Messages are sent to timbreID informing it how to cluster the training data.
• A message to set the k-NN value is sent to timbreID.
• Send each instance again but this time compare the output from timbreID with the correct
label. In order to make this method wait for timbreID’s answer, a flag is set to 0 and, while
this flag is not changed, the thread is put to sleep.
• A message to reset timbreID is sent.
The feature that best improves the classification is added to the selected features list. If a feature
is in this list, every time features are sent to timbreID, the values corresponding to the instance of
the said feature is also sent. The improvement of the iteration is calculated and, if it is bigger than
0, another feature can be added and the cycle repeats itself. Otherwise, the object is considered
trained, messages are sent in order to train timbreID with the selected features and with the cluster
information.
If an integer is received in the right inlet, the inlet to which the timbreID output is connected,
the flag is set to 1 and the “id” method can stop the while cycle.
When a “debug” message is received, the debug flag is set to 1. This is used for debugging
proposes and prints to Max window important log messages.
The “clear” message resets all the class variables to their initial values and a reset message is
sent to timbreID.
If the “knn (int)” message is received, the k-NN value is set to the one specified in the message.
The final method handles the assist message. This is used to provide visual information in the
Max patch window related to what each input and output does.
4.3 User Interface
Due to the constrains imposed by the Max for Live toolkit, in order to have an instrument capable
of outputting MIDI notes, two devices have to be used. The first one is a Max for Live audio
effect, while the second one is a MIDI effect. The user should load both devices in Ableton Live,
LVT loaded as an audio effect (in an effect rack or on an audio track) and the LVT Receiver loaded
in a MIDI track. The audio effect, LVT, receives the audio input and sends messages containing
the transcribed MIDI notes to the MIDI effect, LVT Receiver, which outputs these notes to the
Ableton Live track. These objects’ user interfaces and how they should be used are described in
the following subsections.
4.3.1 LVT
The user interface of the LVT device contains two panels as can be seen in Figure 4.4. In the first
panel, the number of times that each drum vocalisation to train the identifier is repeated and their
corresponding MIDI note are set and a light for each drum sound is displayed. The second panel
4.3 User Interface 25
contains the Train button to initialise the system’s training, the Reset button that resets the system
to its initial state, a light that informs if the system is trained or not and the device identifier.
Figure 4.4: User interface of the LVT device
The first step when using this system is to set the number of repetitions for each vocalisation.
If one of the percussive instruments is not going to be vocalised, setting this value to 0 will disable
it. Then, the desired MIDI notes for each vocalisation should be set. These should be the notes
that correspond to the desired sounds in the instrument that follows the LVT receiver in the MIDI
chain. LVT allows for up to five different drum types.
To start the training of the system starts by pressing the Train button in the second panel. The
user should then begin vocalising the desired drum sounds by repeating each sound the number
of times defined earlier. As each vocalisation is detected, the correspondingly labelled drum type
will light up. When the training phase has finished, the Trained light is turned on.
Now, the system is trained, or it can be reset to its initial state by pressing the reset button.
When a vocalisation is performed, the corresponding light will flash. The messages are not yet
reaching the LVT Receiver. The set up of the LVT Receiver is described in the next subsection.
In order to link the interface with the transcription system, several small Max patches were
used. The train button works as switch for the audio input. A patch was created to count the num-
ber of training instances and to label each one appropriately. One patch calculates the velocity of
the detected event from the instantaneous RMS energy value. Another patch packs the classifica-
tion from timbreID as a MIDI note and sends it to the LVT Receiver. The final patch is responsible
for getting the device identifier from Ableton Live and to present it in the LVT window.
4.3.2 LVT Receiver
The LVT Receiver user interface only has one panel. This panel contains a drop-down menu used
to select the identification of the LVT sender device, a reset button and a mute button, used to stop
listening the the messages from LVT.
The setup of the LVT receiver is simple. The identifier that is displayed in the LVT window
should be selected from the LVT receiver drop-down menu. After this step, the LVT system is
ready to use and a VST or a plugin can be loaded in Ableton Live after the LVT receiver. The
26 Methodology
Figure 4.5: User interface of LVT receiver device
MIDI transcription can be stored in an Ableton Live MIDI clip by creating a new MIDI track that
gets its MIDI input from the Post FX channel of the track where the LVT Receiver is loaded. There
is a constant delay between the vocalisation and the transcription, therefore, the start time of the
MIDI clip should be adjusted manually when the recording is finished.
This receiver device is responsible for unpacking the messages received from the chosen LVT
device and converting them to MIDI notes. The message is formatted using the midiformat object
which packs the information in a MIDI message that is then output to Ableton Live.
Chapter 5
Data Preparation
In this chapter, the approach used to collect a dataset of vocalised percussion will be described
in detail. The method used to record the vocalisations will be explained first, followed by the
procedure used to annotate these recordings.
5.1 Dataset Recording
In order to collect different vocalisations of percussion, a group of 11 men and 9 women were
selected. A similar gender distribution was desired so that this dataset samples the real world.
Almost all of the participants work at a University Radio and 7 of the participants are music en-
thusiasts and have some knowledge of music making, while the rest have basic music knowledge.
Only one of the people involved has beatboxing skills, while the others vocalised the percussive
sounds in a “less professional” way.
Participants were first asked to reproduce 4 bars of a fixed drum pattern with the vocalisations
they were more comfortable with and, after this, 4 bars of improvisation using the same sounds.
The pattern the participants were asked to reproduce was a simple 4 bar loop with kicks on the first
and second beat, snare on the third and hi-hats between them, as shown in Figure 5.1. Participants
were first familiarised with this pattern by listening to it played through an 808 drum-kit, as some
of them could not read music scores. Based on the state of the art, only these three drum sounds
were chosen to be vocalised.
Figure 5.1: Pattern participants were asked to reproduce
27
28 Data Preparation
Participants were given Audio Technica ATH M50X headphones with a metronome at 140
bpm to have a time reference. In order to collect data for possible different use cases, the audio
output was recorded through 3 different microphones (one from a laptop, an AKG c4000b and one
from an iPad). The first one had a lot of noise due to its poor quality, the second one provided a ref-
erence of a good microphone, while the last one is a reference to good mobile phone microphone.
The recordings were made in a sound treated studio in order to isolate external noises.
This process led to 120 audio clips with approximately 6 seconds of duration. The number of
different drum hits contained in these recordings can be seen in Table 5.1.
Fixed Pattern Improvisation TotalKick 8×20×3 = 480 164×3 = 492 972Snare 4×20×3 = 240 98×3 = 294 534Hi-hat 8×20×3 = 480 181×3 = 543 1023
Table 5.1: Number of individual hits contained in the recordings
5.2 Dataset Annotation
The audio files that resulted from the recordings referred in the previous section were compiled in
2 folders, one for the improvisation part and the other one for the fixed pattern recordings. Each
file was given a name that contained the code for each person, ‘I’ if it was an improvisation or ‘P’
if it was a fixed pattern and a number corresponding to the microphone that recorded the audio (1
for laptop microphone, 2 for the AKG microphone and 3 for the iPad one).
Besides saving the recordings in files, these were also collected in an Ableton Live project, as
seen in Figure 5.2. The files were split in 2 groups, each one having a track for each microphone.
Both the files with the annotations and the Ableton Live project are available in the dissertation
website1.
The annotation of the audio was done using Sonic Visualizer2, an application for viewing and
analysing the contents of music audio files, developed at the Centre for Digital Music, Queen
Mary University of London. The detection of the onsets was done manually and each event was
labelled as kick, snare or hi-hat. The transcription was both saved as a .csv file and as a MIDI file,
with the kick in the note 36, snare being the note 38 and the hi-hat 42. The resulting files were
named with the same name as the audio files, but without the microphone number. An example of
a transcription can be seen in Figure 5.3.
From the resulting transcriptions, we could see that participants did not reproduce the hits on
time as per the 140bpm metronome, as seen on Figure 5.4. However, this has no effect on the
transcription system or accuracy which is not tempo dependent.
Participants did not vocalise the drum hits in the same way. First, the user with beatboxing
knowledge vocalised the kick and snare in a different manner than the rest of the participants, as
1https://lvtsmc.wordpress.com/2http://www.sonicvisualiser.org/
5.2 Dataset Annotation 29
Figure 5.2: Organization of the dataset files in an Ableton Live project
can be heard in the dataset (JSil audio clip). Besides this, participants vocalised the kick and the
snare in the way it was easiest for them, therefore, different sounds were used to reproduce the
same drum hits, as can be seen in Figure 5.5 and 5.6.
30 Data Preparation
Figure 5.3: Example the audio annotation in Sonic Visualizer
Figure 5.4: Example of how participants vocalised the pattern
(a) A vocalised kick
(b) A kick reproduced by a beatboxer
Figure 5.5: Two different vocalisations of kick drum
(a) One vocalisation of a snare (b) Another possible vocalisation of a snare
Figure 5.6: Two different vocalisations of a snare drum
Chapter 6
Evaluation
This chapter describes the methodology used to test and evaluate the performance of the LVT
system, in comparison with the existing solutions, LDT [2] and Ableton Live Convert Drums to
MIDI function.
6.1 Experiment Design
The evaluation for LVT comprises one principal experiment that serves two purposes. The first
is to understand how a user specific trained system performs compared to the state of the art, i.e.,
systems which are trained to work on general drum timbres, while the second purpose is to explore
whether LVT can help to improve a producer’s workflow by examining the effort required to get
from a vocalised input pattern to a accurate MIDI representation.
In order to evaluate the three systems in the same data and still being able to use different data
for the training and the evaluation of LVT, some work was done on the dataset. Five kick, snare
and hi-hat vocalisations were extracted from the improvisation part of the dataset so as to create
training clips for the LVT (presented in Section 4.3). These clips were created in a manner that
simulates how a user would train the algorithm, with a speed similar to the one each participant had
in their improvisation audio recording. Seven of the contributors of the dataset did not vocalise at
least five times each drum sound and, therefore, their recordings were removed from the evaluation
data. This resulted in an evaluation set of 13 participants with both a training and a testing audio
clip recorded in three different microphones, which corresponds to 78 audio clips. These clips
were compiled in an Ableton Live project, which is available in the dissertation website1 and that
can be seen in Figure 6.1. This Ableton Live project contains an audio track and three MIDI
tracks for each microphone. The audio tracks contain the training and the testing audio clips and
the MIDI tracks contain the clips with the transcriptions from each system: LVT, LDT Max for
Live device [2] and Ableton Live Convert Drums to MIDI.
To obtain a measure of the accuracy of a user trained system compared to the state of the art
systems, the F-measure of the transcriptions was calculated. The F-measure is the harmonic mean
1https://lvtsmc.wordpress.com/
31
32 Evaluation
Figure 6.1: Ableton project for the evaluation
of precision and recall, and can be calculated as follows:
2∗ precision∗ recallprecision+ recall
. (6.1)
Where:
precision =truepositives
truepositives+ f alsepositives(6.2)
recall =truepositives
truepositives+ f alsenegatives. (6.3)
This was calculated by importing all the transcription MIDI clips with MIDI Tools2 into MAT-
LAB, comparing the transcriptions with the annotations and then calculating the F-measure for
each drum and the average of these values. These results were plotted to see the effect of increas-
ing the F-measure tolerance window (as a means to understand the effect of temporal localisation)
in this accuracy measurement.
Finally, to acquire a measure of how this system can improve a producers workflow, the time
to get a transcription and the number of operations needed to get to the desired patterns were
calculated.
To measure the time an Ableton Live transcription takes, a stopwatch was used to measure the
time since the "Convert Drums to MIDI" button was pressed until the resulting MIDI clip came
into view. This procedure was done in 9 random clips from the 3 different microphones. These
measurements were then averaged. As LDT works in real time, the time to achieve a transcription
is the same as the time of the audio recording to be transcribed. Finally, in order to obtain a
2http://www.ee.columbia.edu/∼csmit/matlab_midi.html
6.2 Results 33
transcription from LVT, two times have to be measured. First, the training time corresponds to the
time of the training audio clips which was calculated and averaged. The time to transcribe a given
audio recording when the system is trained corresponds to the time of this recording.
Then, the number of operations to achieve the desired pattern, that can be seen in Figure 6.2
was calculated.
Figure 6.2: Desired Pattern
Figure 6.3: How the number of operations was calculated. 1) Delete the extra events; 2) Correctthe events that can be corrected; 3) Add the missing events.
The possible operations are divided in three categories: to correct, remove, or add an event.
The procedure to compute these values is explained in Figure 6.3. First, all the additional events
are removed. These are the events that do not correspond to a real onset or that cannot be corrected
to the real classification. Afterwards, the events that are the result of a misclassification are cor-
rected and, finally, the missing events are added to the MIDI clip. The number of operations for
each clip was written down and, then, the number of operations for each category was calculated
for each microphone and for each transcription system. Since it is not reasonable to assume that
a producer editing the transcriptions would work at a constant speed, it was deemed more reliable
to take an objective measurement of workflow effort in terms of the number of operations, rather
than recording the temporal duration - as per the algorithm processing.
6.2 Results
In this section the results from the evaluation previously described are presented.
The calculated results of the F-measure accuracy (with a tolerance window of ± 0.035 sec-
onds) for each microphone are presented in Tables 6.1, 6.2 and 6.3.
34 Evaluation
Kick Snare Hi-hatLVT 27.9% 18.1% 7.6%Ableton Live 34.4% 34.3% 15.8%LDT 3.87% 15.4% 11.8%
Table 6.1: F-measure results for the PC microphone
Kick Snare Hi-hatLVT 91.4% 69.1% 80.2%Ableton Live 51.8% 47.0% 29.7%LDT 53.8% 20.4% 41.9%
Table 6.2: F-measure results for the AKG microphone
Kick Snare Hi-hatLVT 82.2% 39.5% 70.6%Ableton Live 55.2% 43.0% 31.8%LDT 58.4% 22.3% 42.8%
Table 6.3: F-measure results for the iPad microphone
These results show that, except for the laptop microphone and the snare from the iPad micro-
phone, LVT achieves much better performance than the other systems, sometimes even the double
of the F-measure is achieved. All systems report low accuracy on the laptop microphone. The
performance of the state of the art systems in the AKG and iPad microphone are similar, whilst,
for LVT, the use of the laptop microphone leads to the worst overall performance.
The effect of changing the window size value for the F-measure for every drum can be seen
in Figure 6.4, where each line represents the performance of each system for each microphone (1-
laptop microphone, 2- AKG microphone and 3-iPad microphone).
The values for the F-measure stay approximately constant for values higher than 0.035 sec-
onds. The performance of LVT on the recordings from the AKG microphone surpasses the others
by a significant amount, as shown by the fact that it outperforms all other configurations for all
window sizes. On the kick and hi-hat, it is followed by the performance of LVT on the iPad
recordings.
In order to see the effect of user-specific training on the performance of LVT, an example is
provided where LVT is trained on one user and tested on another – and vice-versa. When training
the LVT with a different person with different vocalisations, the accuracy of the transcription
is decreased as can be seen in Figures 6.5 and 6.6. In the upper part of the pictures there is the
transcription of the user when trained with its own vocalisations, while the bottom part corresponds
to the transcription when trained with the other user. As has been pointed out by these figures,
without the user-specific training, a lot of misclassifications occur.
A surprising observation for these two users is that the feature selection in the training phase
suggested the need only for a single feature per user. In other words, the different drum vocalisa-
tions in training could be perfectly separated using just one dimension. The selected feature for
6.2 Results 35
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
0.2
0.4
0.6
0.8
1Kick
Window Size in seconds
Fm
easu
re
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
0.2
0.4
0.6
0.8
1Snare
Window Size in seconds
Fm
easu
re
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
0.2
0.4
0.6
0.8
1Hi hat
Window Size in seconds
Fm
easu
re
LDT1LDT2LDT3ABL1ABL2ABL3LVT1LVT2LVT3
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
0.2
0.4
0.6
0.8
1Average
Window Size in seconds
Fm
easu
re
Figure 6.4: Effect of changing the window size per vocalised drum sounds and across micro-phones. All LDT scores are shown in red, Ableton Live (ABL) in green and LVT in blue. Thesolid lines indicate the laptop microphone, the dotted lines the AKG microphone, and the dashedlines the iPad microphone.
Figure 6.5: Transcription of the first user vocalisations using the LVT system trained by the seconduser
the first user was the spectral flatness from 0 to 250 Hz and for the second one was the spectral
skewness.
The effect of selecting the wrong feature on clustering can be seen in Figures 6.5, 6.6 and 6.7.
In the latest, the distribution of kicks (circle), snares (diamond) and hi-hat (*) are shown. In a),
the second user vocalised pattern elements are distributed from 0 to 1 according to the value of
the selected feature (skewness). In b), the elements from the first user vocalisation according to
36 Evaluation
Figure 6.6: Transcription of the second user vocalisations using the LVT system trained by thefirst user
the feature selected from the second user are displayed and finally, in c) these same elements are
distributed according to the most appropriate feature according to SFS (flatness from 0 to 250 Hz).
When the appropriate feature is selected, it can be seen that the different drum hits are closely
clustered. When this is not the case, the drum hits are more spread and the regions for each drum
are not well defined.
Figure 6.7: Effect of choosing a wrong feature for a user. a) 2nd user; b)1st user with feature for2nd user; c)1st user.
In terms of a qualitative comparison between LVT, LDT and Ableton Live, an example tran-
scription can be seen in Figure 6.8 for LVT; for Ableton Live Convert Drums to MIDI in 6.9; and
for LDT in 6.10.
For an example where LVT transcribes the vocalised pattern accurately, Ableton Live detects
constantly hi-hats on top of the other drum sounds and even on silence. Furthermore, it did not
detect any kick drum in this recording. In turn, LDT, besides detecting all the ground truth events,
also identified a lot of false positives.
6.2 Results 37
Figure 6.8: Example of an LVT transcription
Figure 6.9: Example of an Ableton Live Convert Drums to MIDI transcription
Figure 6.10: Example of a LDT transcription
38 Evaluation
The timing measurements for each of the systems are the following:
• Ableton Live Convert Drums to MIDI: 12.9s
• LVT: 6.2s+6.9s= 13.1s (average of training clips + audio clip time)
• LDT: 6.9s (audio clip time)
In order to achieve a transcription, LDT is the quickest one, followed by Ableton Live and then
LVT. Ableton Live Convert Drums to MIDI function firstly shows a loading window, but after it
is finished, some time elapses until this DAW displays the MIDI clip in its user interface.
In addition to these processing times, required to give an initial automatic transcription, Tables
6.4, 6.5 and 6.6 summarise the results obtained from counting the total number of operations
needed to obtain the desired pattern for each microphone. The total number of events for each
microphone is 13∗20 = 260 which corresponds to the number of users ∗ the number of events in
each audio clip.
Corrected Added RemovedLVT 26 182 3Ableton Live 22 33 440LDT 52 124 95
Table 6.4: Number of Operations for the PC microphone
Corrected Added RemovedLVT 39 7 15Ableton Live 33 12 296LDT 52 24 206
Table 6.5: Number of Operations for the AKG c4000b microphone
Corrected Added RemovedLVT 57 40 5Ableton Live 67 10 215LDT 51 22 198
Table 6.6: Number of Operations for the iPad microphone
From these tables, it is easy to see that the transcriptions from LDT and Ableton Live require
a lot of events to be removed, whilst the ones from LVT do not. The number of corrected vocalisa-
tions from LVT and Ableton are similar, while the ones from LDT remain approximately constant
for all microphones. For the laptop and iPad microphones, LVT under-detected more events than
the rest of the systems.
6.3 Discussion 39
6.3 Discussion
In this section, the results presented in the previous section are analysed and discussed.
By examining the previously shown results provided in Tables 6.1, 6.2 and 6.3, we can under-
stand that the LVT provides a transcription closer to the ground truth than the generally trained
state of the art systems, shown by the higher F-measure. Besides the detail that LVT is trained
per user, these results may derive from the fact that this system does not try to detect polyphonic
events (more than one drum vocalisation at the same time) as the other systems do. Furthermore,
LVT does not detect as many events as the other systems, and, therefore, this has an influence in
the F-measure results, in terms of false positives.
From Figure 6.7, we can see that feature selection is an important step to acquiring an accurate
transcription. In order to have the k-NN algorithm work as good as possible, the different vocali-
sations must be tightly clustered, as it can be seen in a) and c). From Figures 6.5 and 6.6 we can
understand that having a system trained with a different user reports an inaccurate classification
and, therefore, we can see the importance of training the system to adapt the user vocalisations.
For the small cost in terms of timing for training the LVT, the transcription accuracy is greatly
increased and, as shown by having far fewer post-transcription operations, a significant amount of
time is saved when correcting the transcribed pattern. On this basis, the end-to-end workflow, from
training to transcription to correction is most efficient for LVT suggesting a real tangible benefit for
user-adaptive analysis. However, while LVT performs especially well with the AKG microphone
and with the iPad microphone, its performance with the computer microphone is particularly poor.
This microphone has a lot of background noise and the system is not able to detect onsets and
hence cannot provide a transcription. Thus within the processing pipeline, accurate onset detection
is extremely important, and its impact is directly observable in the transcription accuracy.
Concerning other possible limitations of LVT, if a user does not vocalise the drum sounds the
same way in the training and in the identification phase, the transcription will not work well – a
factor which more generally trained systems would not be susceptible to. Furthermore, if a user
vocalises drums that sound too similar to each other, as it was the case in some of the clips from the
dataset, there will not be a separation clear enough of the events from the perspective of the audio
features, and therefore the machine learning component will struggle to identify them correctly.
Finally, if a user vocalises the drum sounds too quietly, the onset detection will not work and the
event will not be transcribed.
As a final point, in order to have a quantitative measure on how a system performs in terms of
producer’s workflow, the number of operations to achieve a desired pattern is considered a more
meaningful measure than the F-measure. This is due to the fact that F-measure is dependent on
the window size and on the fact that a misclassification is represented both as a false positive and
as a false negative. The same occurs with poorly timed transcriptions, where an onset outside the
F-measure window also contributes to this calculation. These two possible deviations are easily
fixed in a DAW via simple shifting operations and are thus, less significant errors than totally
spurious false positives or totally missing events.
40 Evaluation
Chapter 7
Conclusions
7.1 Summary
In this dissertation, a new interface for music creation, called LVT, was presented. This system
allows Ableton Live users to sequence MIDI patterns that can be used for designing rhythms by
using their voice. The state of the art systems, including one already in Ableton Live, are not
able to transcribe vocalised percussion effectively, as these are trained for general recorded drum
sounds which are not vocalised. Different people vocalise drum sounds in different manners, a
snare drum vocalisation from a user can sound similar to the hi-hat vocalisation from another user.
LVT has to be trained before it is used, in order to fit the vocalisations of any end user. As each
user can choose the desired vocalisations for each drum sound, the system is versatile enough to
also transcribe drums or any kind of unpitched percussive sounds. As long as the training sounds
are different enough from each other, the system is able to choose the features that provide a
good separation and therefore a good classification accuracy for any input. In order to improve
the accuracy of this system, a Max external that implements the Sequential Forward Selection for
selecting features was developed. LVT is implemented as a Max for Live device, which enables
Ableton Live users to use this system by interacting with the simple and easy-to-use graphical user
interface designed for it.
The evaluation of the LVT and of the existing state of the art systems was done by running tests
on a dataset that was recorded and annotated. In order to collect different percussion vocalisations,
partakers expressed a fixed pattern with the vocalisations they found more suitable. The evaluation
of the accuracy of the transcription was done by calculating the F-measure and by counting the
number of actions needed to transform the resulting transcription in the desired pattern. The F-
measure is an adequate evaluation of the transcription accuracy while the number of operations
relates to how this accuracy affects the Ableton Live user workflow. LVT produced superior results
in both tests, showing that this tool can be used as an alternative to the existing drum transcription
systems in order to create MIDI drum patterns using the voice as the instrument.
41
42 Conclusions
7.2 Future Work
Despite the good results of the LVT described earlier, there are some features that can be added
to this system in order to improve its usability and performance. Due to time restrictions and the
amount of work some of these features need, they were not yet implemented. These improvements
are the following:
• Settings window: Add a window to the User Interface where system parameters can be set.
Examples of possible relevant parameters that may be changed by the user are the number
of neighbours for the k-NN classifier and the threshold and mode for the onset detection.
These values can only be changed inside the Max for Live patch and, therefore, they are not
easily accessible to the user.
• Save and Load button: Adding a Save and a Load button. This enables the possibility to
save the training in a file, so that the system does not have to be trained each time the user
loads it and provides more portability between computers for the same user.
• More feature selection methods and machine learning algorithms: Evaluate the effec-
tiveness of other feature selection or extraction methods and add the possibility to choose
the one desired by the user. Possible methods to be added are Sequential Backward Selec-
tion, Generalised Sequential Forward and Backward Selection, Sequential Forward Floating
Search or even adding the possibility for the user to manually select the desired features to
be extracted. Another machine learning algorithms can also be implemented and evaluated.
• Export the selected features report: The system prints to the Max command window the
index of the selected features. Adding the possibility of exporting a report that contains the
chosen features is a possible improvement.
• More features extracted: Adding more features to the feature extraction module. Temporal
and spectral features can be added such as the duration or the spectral roughness. Adding
the duration as a feature can help to make a distinction between open and closed hi-hats, as
some vocalisations of these cymbals only differ in their duration.
• Further Testing: More testing should be done on LVT with a different number of vocalised
drum sounds and with different values for the training instances.
• Pattern-Based Analysis: Researching pattern-based analysis techniques to be incorporated
in the LVT system. A possible implementation of this topic is to detect microtimings on the
vocalised pattern and correct the result when these are not present.
7.3 Perspectives on the Project 43
7.3 Perspectives on the Project
To introduce participants with no background in music production to music creation and its tech-
nology, as well as allowing them to improvise with vocalised drum sounds, was a rewarding expe-
rience. Furthermore, this project provided me with insight into the development of Max externals
and how Max MSP works, on feature extraction, on onset detection and on machine learning.
44 Conclusions
Appendix A
seqfeatsel C Code
This appendix contains the C code for the Max external seqfeatsel and the corresponding flowchart.
45
46 seqfeatsel C Code
A.1 seqfeatsel Code
1 # i n c l u d e " e x t . h " / / s t a n d a r d Max i n c l u d e , a lways r e q u i r e d2 # i n c l u d e " e x t _ o b e x . h " / / r e q u i r e d f o r new s t y l e Max o b j e c t3 / / / / / / / / / / / / / / / / / / / / / / / / / / o b j e c t s t r u c t4
5 t y p e d e f s t r u c t i n s t a n c e6 {7 do ub l e ∗ i n s t a n c e ;8 } t _ i n s t a n c e ;9
10 t y p e d e f s t r u c t member11 {12 i n t ∗member ;13 } t_member ;14
15 t y p e d e f s t r u c t _ s e q f e a t s e l16 {17 t _ o b j e c t ob ; / / t h e o b j e c t i t s e l f ( must be f i r s t )18 boo l i d e n ; / / i s i t a l r e a d y t r a i n e d o r n o t ?19 boo l f l a g ; / / has t i m b r e I D g i v e n an answer ?20 boo l debug ; / / used f o r debugg ing p u r p o s e s21 boo l f a s e ; / / i s t i m b r e I D answer a no c a r e ?22 l ong numFea tu res ; / / number o f f e a t u r e s r e c e i v e d23 l ong rowCount ; / / c o u n t s t h e rows24 l ong u l t i m a N o t a ; / / l a s t r e c e i v e d MIDI n o t e25 l ong r e s p o s t a ; / / answer from t i m b r e I D26 l ong knn ; / / knn v a l u e27 l ong numNotas ; / / t o t a l number o f n o t e s28 s h o r t nSe l ; / / number o f s e l e c t e d f e a t u r e s29 t_member ∗ n o t a s P C l u s t e r ; / / a r r a y wi th t h e n o t e s f o r t h e c l u s t e r message30 t _ i n s t a n c e ∗ t r a i n i n g T a b ; / / a r r a y wi th t h e r e c e i v e d f e a t u r e s31 i n t ∗ s e l C o l ; / / columns s e l e c t e d t h r o u g h s f s32 vo id ∗ a _ o u t ; / / o u t p u t t h e column t o use33 vo id ∗ b_ou t ; / / o u t p u t s t h e rows f o r t h e kNN e x t e r n a l34 } t _ s e q f e a t s e l ;35
36 / / / / / / / / / / / / / / / / / / / / / / / / / f u n c t i o n p r o t o t y p e s37 / / / / s t a n d a r d s e t38 vo id ∗ s e q f e a t s e l _ n e w ( t_symbol ∗s , l ong argc , t_a tom ∗ a rgv ) ; / / o b j e c t c r e a t i o n
method39 vo id s e q f e a t s e l _ a s s i s t ( t _ s e q f e a t s e l ∗x , vo id ∗b , l ong m, long a , c h a r ∗ s ) ;40 vo id s e q f e a t s e l _ f r e e ( t _ s e q f e a t s e l ∗x ) ;41
42
43 vo id s e q f e a t s e l _ m e s s a g e ( t _ s e q f e a t s e l ∗x , t_symbol ∗s , l ong argc , t_a tom ∗ a rgv ) ;44 vo id s e q f e a t s e l _ i n 1 ( t _ s e q f e a t s e l ∗x , l ong e n t r a d a ) ;45 vo id s e q f e a t s e l _ d e b u g ( t _ s e q f e a t s e l ∗x ) ;46 vo id s e q f e a t s e l _ i d ( t _ s e q f e a t s e l ∗x ) ;
A.1 seqfeatsel Code 47
47 vo id s e q f e a t s e l _ c l e a r ( t _ s e q f e a t s e l ∗x ) ;48 vo id s e q f e a t s e l _ p r i n t ( l ong argc , t_a tom ∗ a rgv ) ;49 vo id s e q f e a t s e l _ p r i n t 2 ( l ong c e r t a , l ong p r o p o s t a , l ong maxrows , l ong j ) ;50 vo id s e q f e a t s e l _ k n n ( t _ s e q f e a t s e l ∗x , l ong knn ) ;51
52 / / / / / / / / / / / / / / / / / / / / / / / / g l o b a l c l a s s p o i n t e r v a r i a b l e53 vo id ∗ s e q f e a t s e l _ c l a s s ;54
55
56 vo id ex t_main ( vo id ∗ r )57 {58 t _ c l a s s ∗c ;59
60 c = c l a s s _ n e w ( " s e q f e a t s e l " , ( method ) s e q f e a t s e l _ n e w , ( method ) s e q f e a t s e l _ f r e e ,( l ong ) s i z e o f ( t _ s e q f e a t s e l ) ,
61 0L , A_GIMME, 0) ;62
63 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ m e s s a g e , " l i s t " , A_GIMME, 0) ;64 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ i n 1 , " i n 1 " , A_LONG, 0) ;65 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ i d , " i d " , 0 ) ;66 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ c l e a r , " c l e a r " , 0 ) ;67 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ d e b u g , " debug " , 0 ) ;68 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ k n n , " knn " , A_LONG, 0) ;69 c l a s s _ a d d m e t h o d ( c , ( method ) s e q f e a t s e l _ a s s i s t , " a s s i s t " , A_CANT, 0 ) ;70
71 c l a s s _ r e g i s t e r (CLASS_BOX, c ) ;72 s e q f e a t s e l _ c l a s s = c ;73 }74
75 vo id s e q f e a t s e l _ p r i n t 2 ( l ong c e r t a , l ong p r o p o s t a , l ong maxrows , l ong j )76 {77 p o s t ( " R e s p o s t a c e r t a : %ld , R e s p o s t a TimbreID : %l d , Numero de i n s t a n c e s :
%l d , j : %l d . " , c e r t a , p r o p o s t a , maxrows , j ) ;78 }79
80 vo id s e q f e a t s e l _ k n n ( t _ s e q f e a t s e l ∗x , l ong k nnv a l ) {81
82 x−>knn = k nn va l ;83
84 }85 vo id s e q f e a t s e l _ p r i n t ( l ong argc , t_a tom ∗ a rgv )86 {87 l ong i ;88 t _a tom ∗ap ;89 p o s t ( " t h e r e a r e %l d a rgumen t s " , a r g c ) ;90 / / i n c r e m e n t ap each t ime t o g e t t o t h e n e x t atom91 f o r ( i = 0 , ap = a rgv ; i < a r g c ; i ++ , ap ++) {92 s w i t c h ( a t o m _ g e t t y p e ( ap ) ) {93 c a s e A_LONG:
48 seqfeatsel C Code
94 p o s t ( "%l d : %l d " , i + 1 , a t o m _ g e t l o n g ( ap ) ) ;95 b r e a k ;96 c a s e A_FLOAT :97 p o s t ( "%l d : %.2 f " , i + 1 , a t o m _ g e t f l o a t ( ap ) ) ;98 b r e a k ;99 c a s e A_SYM:
100 p o s t ( "%l d : %s " , i + 1 , atom_getsym ( ap )−>s_name ) ;101 b r e a k ;102 d e f a u l t :103 p o s t ( "%l d : unknown atom t y p e (% l d ) " , i + 1 , a t o m _ g e t t y p e ( ap ) ) ;104 b r e a k ;105 }106 }107 }108
109 vo id s e q f e a t s e l _ m e s s a g e ( t _ s e q f e a t s e l ∗x , t_symbol ∗s , l ong argc , t_a tom ∗ a rgv ) {110
111 i f ( x−>i d e n == f a l s e ) {112 i n t i , l i s t L e n g t h ;113 l ong l i n h a = x−>rowCount ;114 l i s t L e n g t h = a r g c ;115
116 x−> t r a i n i n g T a b = ( t _ i n s t a n c e ∗ ) s y s m e m _ r e s i z e p t r ( x−>t r a i n i n g T a b ,( x−>rowCount + 1) ∗ s i z e o f ( t _ i n s t a n c e ) ) ;
117 x−> t r a i n i n g T a b [ l i n h a ] . i n s t a n c e = ( d ou b l e ∗ ) sysmem_newptr ( l i s t L e n g t h ∗s i z e o f ( d ou b l e ) ) ;
118
119 x−>rowCount ++;120 i f ( x−>debug == 1) { p o s t ( " Receb i Mensagem e e n t r e i no i d e n == f a l s e " ) ; }121 i f ( x−>numFea tu res != l i s t L e n g t h ) {122 x−>s e l C o l = ( i n t ∗ ) s y s m e m _ r e s i z e p t r ( x−>s e l C o l , l i s t L e n g t h ∗ s i z e o f ( i n t ) ) ;123 x−>numFea tu res = l i s t L e n g t h ; }124 i f ( x−>debug == 1) { p o s t ( " a r g c = %l d e numFea tu res = %l d " , a rgc ,
x−>numFea tu res ) ; }125
126 i f ( l i n h a == 0) { / / F i r s t Note127 i f ( x−>debug == 1) { p o s t ( " P r i m e i r a Nota " ) ; }128
129 x−>u l t i m a N o t a = a t o m _ g e t l o n g ( a rgv ) ;130 i f ( x−>debug == 1) { p o s t ( " u l t i m a n o t a = %l d " , x−>u l t i m a N o t a ) ; }131 x−>numNotas = 0 ;132 x−>n o t a s P C l u s t e r = ( t_member ∗ ) s y s m e m _ r e s i z e p t r ( x−>n o t a s P C l u s t e r ,
s i z e o f ( t_member ) ) ;133 x−>n o t a s P C l u s t e r [ 0 ] . member = ( i n t ∗ ) sysmem_newptr (3 ∗ s i z e o f ( i n t ) ) ;134 x−>n o t a s P C l u s t e r [ 0 ] . member [ 0 ] = 0 ;135 x−>n o t a s P C l u s t e r [ 0 ] . member [ 1 ] = 0 ;136 }137
A.1 seqfeatsel Code 49
138 i f ( x−>u l t i m a N o t a != a t o m _ g e t l o n g ( a rgv ) ) { / / i f t h e r e c e i v e d n o t e i sd i f f e r e n t from t h e p r e v i o u s one
139 i f ( x−>debug == 1) { p o s t ( " Nota D i f e r e n t e " ) ; }140
141 x−>u l t i m a N o t a = a t o m _ g e t l o n g ( a rgv ) ;142
143 x−>n o t a s P C l u s t e r [ x−>numNotas ] . member [ 2 ] = l i n h a − 1 ; / / end of t h ep r e v i o u s row
144
145 x−>numNotas ++;146 x−>n o t a s P C l u s t e r = ( t_member ∗ ) s y s m e m _ r e s i z e p t r ( x−>n o t a s P C l u s t e r ,
( x−>numNotas + 1) ∗ s i z e o f ( t_member ) ) ;147 x−>n o t a s P C l u s t e r [ x−>numNotas ] . member = ( i n t ∗ ) sysmem_newptr (3 ∗
s i z e o f ( i n t ) ) ;148
149 x−>n o t a s P C l u s t e r [ x−>numNotas ] . member [ 0 ] = x−>numNotas ; / / new n o t e150 x−>n o t a s P C l u s t e r [ x−>numNotas ] . member [ 1 ] = l i n h a ; / / s t a r t i n d e x o f new
n o t e151 }152 x−> t r a i n i n g T a b [ l i n h a ] . i n s t a n c e [ 0 ] = x−>numNotas ;153 i f ( x−>debug == 1) { p o s t ( " [% l d ][% l d ] : %l d " , 0 , l i n h a , x−>numNotas ) ; }154
155 f o r ( i = 1 ; i < l i s t L e n g t h ; i ++)156 x−> t r a i n i n g T a b [ l i n h a ] . i n s t a n c e [ i ] = a t o m _ g e t l o n g ( a rgv + i ) ;157 }158
159 i f ( x−>i d e n == t r u e ) {160
161 t _a tom ∗ s a i d a = ( t_a tom ∗ ) sysmem_newptr ( x−>nSe l ∗ s i z e o f ( t_a tom ) ) ;162 / / f i l t e r s columns163 i f ( x−>debug == 1) { p o s t ( " x i d e n e v e r d a d e " ) ; }164
165 f o r ( i n t k = 0 ; k < x−>nSe l ; k ++) {166 a t o m _ s e t f l o a t ( s a i d a + k , a t o m _ g e t f l o a t ( a rgv + x−>s e l C o l [ k ] ) ) ;167 }168 o u t l e t _ l i s t ( x−>b_out , NULL, x−>nSel , s a i d a ) ;169 s y s m e m _ f r e e p t r ( s a i d a ) ;170
171 }172 }173
174 vo id s e q f e a t s e l _ i d ( t _ s e q f e a t s e l ∗x ) {175
176 s h o r t i , j , k , improv , numCorrec tasOld , l , numCorrec tas , c , maximum ,ind iceMax ;
177 numCorrec tasOld = 0 ;178
179 t _a tom ∗mensknn = ( t_a tom ∗ ) sysmem_newptr ( s i z e o f ( t_a tom ) ) ;180 t _a tom ∗m e n s C l u s t e r = ( t_a tom ∗ ) sysmem_newptr (4 ∗ s i z e o f ( t_a tom ) ) ;
50 seqfeatsel C Code
181
182 t _a tom ∗ s a i d a = ( t_a tom ∗ ) sysmem_newptr ( 0 ) ;183 t _a tom ∗ t r e i n o = ( t_a tom ∗ ) sysmem_newptr ( 0 ) ;184
185 i n t ∗ c o r r e c t a s = ( i n t ∗ ) sysmem_newptr ( x−>numFea tu res ∗ s i z e o f ( i n t ) ) ;186 t _ symbol ∗ c l u s t e r ,∗ knnsym ;187 c l u s t e r = gensym ( " m a n u a l _ c l u s t e r " ) ;188 knnsym = gensym ( " knn " ) ;189 improv = 1 ;190
191 x−>n o t a s P C l u s t e r [ x−>numNotas ] . member [ 2 ] = x−>rowCount −1; / / marca o f im dat a b e l a
192 i f ( x−>debug == 1) { p o s t ( " Es tou no ID " ) ; }193 i f ( x−>rowCount > 1) {194 w h i l e ( improv > 0) {195
196 s a i d a = ( t_a tom ∗ ) s y s m e m _ r e s i z e p t r ( s a i d a , ( x−>nSe l + 1) ∗s i z e o f ( t_a tom ) ) ;
197 t r e i n o = ( t_a tom ∗ ) s y s m e m _ r e s i z e p t r ( t r e i n o , x−>nSe l ∗ s i z e o f ( t_a tom ) ) ; ;198
199 / / Runs a l l columns e x c e p t t h e one wi th l a b e l s200 f o r ( i = 1 ; i < x−>numFea tu res ; i ++) {201 i f ( x−>debug == 1) { p o s t ( " E n t r e i na Coluna %l d " , i ) ; }202
203 / / i f column a l r e a d y s e l e c t e d don t run204 f o r ( k = 0 ; k < x−>nSe l ; k ++) {205 i f ( i == x−>s e l C o l [ k ] ) { c o r r e c t a s [ i ] = 0 ; go to f o r a ; }206
207 }208
209 / / T r a i n s t i m b r e I D one row a t a t ime210 x−>f a s e = f a l s e ;211 f o r ( j = 0 ; j < x−>rowCount ; j ++) {212
213 / / c r e a t e s message and s e n d s i t214 f o r ( k = 0 ; k < x−>nSe l ; k ++) {215 a t o m _ s e t f l o a t ( s a i d a + k , x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ x−>s e l C o l [ k ] ] ) ;216 }217 a t o m _ s e t f l o a t ( s a i d a + x−>nSel , x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ i ] ) ;218 o u t l e t _ l i s t ( x−>a_out , NULL, x−>nSe l + 1 , s a i d a ) ;219
220 i f ( x−>debug == 1) { p o s t ( " E n v i e i p a r a t r e i n a r o t i m b r e I D %f " ,x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ i ] ) ; }
221 }222
223 / / s e n d s c l u s t e r i n g messages224 f o r ( l = 0 ; l < x−>numNotas + 1 ; l ++) {225 a t o m _ s e t l o n g ( mensClus t e r , x−>numNotas + 1) ;226 a t o m _ s e t l o n g ( m e n s C l u s t e r + 1 , l ) ;
A.1 seqfeatsel Code 51
227 a t o m _ s e t l o n g ( m e n s C l u s t e r + 2 , x−>n o t a s P C l u s t e r [ l ] . member [ 1 ] ) ;228 a t o m _ s e t l o n g ( m e n s C l u s t e r + 3 , x−>n o t a s P C l u s t e r [ l ] . member [ 2 ] ) ;229
230 o u t l e t _ a n y t h i n g ( x−>a_out , c l u s t e r , 4 , m e n s C l u s t e r ) ;231 i f ( x−>debug == 1) { p o s t ( " E n v i e i mensagem de C l u s t e r " ) ; }232 }233 a t o m _ s e t l o n g ( mensknn , x−>knn ) ;234 o u t l e t _ a n y t h i n g ( x−>a_out , knnsym , 1 , mensknn ) ;235
236 numCor rec t a s = 0 ;237 x−>f a s e = t r u e ;238 / / s e n d s messages f o r t i m b r e I D t o i d e n t i f y239 f o r ( j = 0 ; j < x−>rowCount ; j ++) { / / f o r e v e r y row240 / / s e n d s row t o o u t p u t 2241 f o r ( k = 0 ; k < x−>nSe l ; k ++) {242 a t o m _ s e t f l o a t ( s a i d a + k , x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ x−>s e l C o l [ k ] ] ) ;243 }244 a t o m _ s e t f l o a t ( s a i d a + x−>nSel , x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ i ] ) ;245 x−> f l a g = f a l s e ;246 o u t l e t _ l i s t ( x−>b_out , NULL, x−>nSe l + 1 , s a i d a ) ;247 i f ( x−>debug == 1) { p o s t ( " E n v i e i p a r a o t i m b r e I D i d e n t i f i c a r %f " ,
x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ i ] ) ; }248 / / w h a i t f o r answer249 w h i l e ( x−> f l a g = f a l s e ) {250
251 s y s t h r e a d _ s l e e p ( 1 ) ;252 }253 i f ( x−>debug == 1) { p o s t ( " S a i do w h i l e " ) ; }254 / / s e e i f answer i s r i g h t255 l ong r e s p o s t a = x−> r e s p o s t a ;256 i f ( r e s p o s t a == ( long ) x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ 0 ] ) {257 / / add t o v e c t o r wi th number o f c o r r e c t answer s258 numCor rec t a s ++;259 i f ( x−>debug == 1) { s e q f e a t s e l _ p r i n t 2 ( ( l ong )
x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ 0 ] , r e s p o s t a , x−>rowCount , j ) ; }260 }261 }262 c o r r e c t a s [ i ] = numCor rec t a s ;263 i f ( x−>debug == 1) { p o s t ( " Coluna %l d tem %l d c e r t a s " , i ,
numCor rec t a s ) ; }264 / / send r e s e t message t o t i m b r e I D265 o u t l e t _ a n y t h i n g ( x−>a_out , gensym ( " c l e a r " ) , 0 , NULL) ;266 f o r a : i f ( x−>debug == 1) { p o s t ( " E n v i e i mensagem de C l e a r " ) ; }267 }268 / / adds t h e column wi th t h e b e s t a c c u r a c y t o t h e s e l e c t e d columns269 maximum = c o r r e c t a s [ 1 ] ;270 ind iceMax = 1 ;271 f o r ( c = 1 ; c < x−>numFea tu res ; c ++)272 {
52 seqfeatsel C Code
273 i f ( x−>debug == t r u e ) { p o s t ( " c o r r e c t a s [% l d ] = %l d e max = %l d " , c ,c o r r e c t a s [ c ] , maximum ) ; }
274 i f ( c o r r e c t a s [ c ] > maximum )275 {276 maximum = c o r r e c t a s [ c ] ;277 ind iceMax = c ;278 }279 }280 / / c a l c u l improv281 improv = maximum − numCorrec tasOld ;282 numCorrec tasOld = maximum ;283 i f ( improv > 0) {284 p o s t ( " Added column %l d " , ind iceMax ) ;285 x−>s e l C o l [ x−>nSe l ] = ind iceMax ;286 x−>nSe l ++;287 }288 i f ( x−>debug == 1) { p o s t ( " Improv %l d " , improv ) ; }289 } / / f im do w h i l e290
291 f o r ( j = 0 ; j < x−>rowCount ; j ++) {292
293 / / c r i a t_a tom [ ] com a l i n h a a e n v i a r e e n v i a294 f o r ( k = 0 ; k < x−>nSe l ; k ++) {295 a t o m _ s e t f l o a t ( t r e i n o + k , x−> t r a i n i n g T a b [ j ] . i n s t a n c e [ x−>s e l C o l [ k ] ] ) ;296 }297 o u t l e t _ l i s t ( x−>a_out , NULL, x−>nSel , t r e i n o ) ;298 i f ( x−>debug == 1) { p o s t ( " TREINEI O TIMBREID" ) ; }299 }300
301 / / C l u s t e r message302 f o r ( l = 0 ; l < x−>numNotas + 1 ; l ++) {303 a t o m _ s e t l o n g ( mensClus t e r , x−>numNotas + 1) ;304 a t o m _ s e t l o n g ( m e n s C l u s t e r + 1 , l ) ;305 a t o m _ s e t l o n g ( m e n s C l u s t e r + 2 , x−>n o t a s P C l u s t e r [ l ] . member [ 1 ] ) ;306 a t o m _ s e t l o n g ( m e n s C l u s t e r + 3 , x−>n o t a s P C l u s t e r [ l ] . member [ 2 ] ) ;307
308 o u t l e t _ a n y t h i n g ( x−>a_out , c l u s t e r , 4 , m e n s C l u s t e r ) ;309 i f ( x−>debug == 1) { p o s t ( " E n v i e i mensagem de C l u s t e r " ) ; }310 }311 a t o m _ s e t l o n g ( mensknn , x−>knn ) ;312 o u t l e t _ a n y t h i n g ( x−>a_out , knnsym , 1 , mensknn ) ;313
314 s y s m e m _ f r e e p t r ( mensknn ) ;315 s y s m e m _ f r e e p t r ( m e n s C l u s t e r ) ;316 s y s m e m _ f r e e p t r ( s a i d a ) ;317 s y s m e m _ f r e e p t r ( t r e i n o ) ;318 s y s m e m _ f r e e p t r ( c o r r e c t a s ) ;319
320 o u t l e t _ a n y t h i n g ( x−>a_out , gensym ( " i d f i n i s h " ) , 0 , NULL) ;
A.1 seqfeatsel Code 53
321 x−>i d e n = t r u e ;322 i f ( x−>debug == t r u e ) { p o s t ( " a c a b e i o c i c l o w h i l e " ) ; }323 }324 }325
326 vo id s e q f e a t s e l _ i n 1 ( t _ s e q f e a t s e l ∗x , l ong e n t r a d a ) {327 / / r e c e i v e s t h e t i m b r e I D g u e s s328 i f ( x−>debug == t r u e ) { p o s t ( " Receb i do t i m b r e I D " ) ; }329
330 i f ( x−>f a s e == t r u e ) {331 i f ( x−>debug == t r u e ) { p o s t ( " Receb i do t i m b r e I D e l i g u e i " ) ; }332 x−> r e s p o s t a = e n t r a d a ;333 x−> f l a g = t r u e ;334 }335 }336
337 vo id s e q f e a t s e l _ a s s i s t ( t _ s e q f e a t s e l ∗x , vo id ∗b , l ong msg , l ong arg , c h a r ∗ d s t )338 { / / a s s i s t message339 i f ( msg == ASSIST_INLET ) {340 s w i t c h ( a r g ) {341 c a s e 0 : s t r c p y ( d s t , " ( l i s t s / messages ) R e c e i v e s f e a t u r e l i s t s w i th t h e
f i r s t e l e m e n t l a b e l i n g t h e e v e n t " ) ; b r e a k ;342 c a s e 1 : s t r c p y ( d s t , " ( i n t e g e r ) R e c e i v e s t h e i d e n t i f i c a t i o n from t h e
c l a s s i f i e r " ) ; b r e a k ;343 }344 }345 e l s e i f ( msg == ASSIST_OUTLET ) {346 s w i t c h ( a r g ) {347 c a s e 0 : s t r c p y ( d s t , " ( l i s t s / messages ) Sends messages and t r a i n i n g l i s t s t o
t h e c l a s s i f i e r " ) ; b r e a k ;348 c a s e 1 : s t r c p y ( d s t , " ( l i s t s ) Sends l i s t s f o r i d e n t i f i c a t i o n as w e l l a s t h e
f i n a l f i l t e r e d l i s t t o t h e c l a s s i f i e r " ) ; b r e a k ;349
350 }351 }352 }353
354 vo id s e q f e a t s e l _ c l e a r ( t _ s e q f e a t s e l ∗x )355 { / / when r e c e i v e s c l e a r356 i f ( x−>debug == 1) { p o s t ( " Receb i C l e a r " ) ; }357
358 x−>i d e n = f a l s e ;359 x−> f l a g = f a l s e ;360 x−>debug = f a l s e ;361
362 x−>numFea tu res = 0 ;363 x−>rowCount = 0 ; / / c o u n t s t h e rows364 x−>u l t i m a N o t a = 0 ;365 x−> r e s p o s t a = 0 ;
54 seqfeatsel C Code
366
367 x−>numNotas = 0 ;368 x−>nSe l = 0 ;369 o u t l e t _ a n y t h i n g ( x−>a_out , gensym ( " c l e a r " ) , 0 , NULL) ;370
371 }372
373 vo id s e q f e a t s e l _ d e b u g ( t _ s e q f e a t s e l ∗x )374 { / / s e t s debug t o 1375
376 x−>debug = t r u e ;377 i f ( x−>debug == 1) { p o s t ( " Receb i debug " ) ; }378
379 }380
381 vo id ∗ s e q f e a t s e l _ n e w ( t_symbol ∗s , l ong argc , t_a tom ∗ a rgv )382 {383 t _ s e q f e a t s e l ∗x = NULL;384
385 x = ( t _ s e q f e a t s e l ∗ ) o b j e c t _ a l l o c ( s e q f e a t s e l _ c l a s s ) ;386
387
388 x−>b_ou t = l i s t o u t ( x ) ;389 x−>a _ o u t = o u t l e t _ n e w ( ( t _ s e q f e a t s e l ∗ ) x , NULL) ;390
391 x−>n o t a s P C l u s t e r = ( t_member ∗ ) sysmem_newptr ( 0 ) ;392 x−> t r a i n i n g T a b = ( t _ i n s t a n c e ∗ ) sysmem_newptr ( 0 ) ;393 x−>s e l C o l = ( i n t ∗ ) sysmem_newptr ( 0 ) ; / / columns s e l e c t e d t h r o u g h s f s394
395 i n t i n ( x , 1 ) ;396
397 x−>i d e n = f a l s e ;398 x−> f l a g = f a l s e ;399 x−>debug = f a l s e ;400 x−>f a s e = f a l s e ;401
402 x−>numFea tu res = 0 ;403 x−>rowCount = 0 ; / / c o u n t s t h e rows404 x−>u l t i m a N o t a = 0 ;405 x−> r e s p o s t a = 0 ;406 x−>knn = 1 ;407
408 x−>numNotas = 0 ;409 x−>nSe l = 0 ;410 r e t u r n x ;411 }412
413 vo id s e q f e a t s e l _ f r e e ( t _ s e q f e a t s e l ∗x ) {414 i n t i ;
A.2 Flowchart of the seqfeatsel external 55
415 i f ( x−>numNotas != 0) {416 f o r ( i = 0 ; i < x−>numNotas + 1 ; i ++)417 s y s m e m _ f r e e p t r ( x−>n o t a s P C l u s t e r [ i ] . member ) ;418 }419 f o r ( i = 0 ; i <x−>rowCount ; i ++)420 s y s m e m _ f r e e p t r ( x−> t r a i n i n g T a b [ i ] . i n s t a n c e ) ;421
422 s y s m e m _ f r e e p t r ( x−>n o t a s P C l u s t e r ) ;423 s y s m e m _ f r e e p t r ( x−> t r a i n i n g T a b ) ;424 s y s m e m _ f r e e p t r ( x−>s e l C o l ) ;425 }
seqfeatselclean.c
A.2 Flowchart of the seqfeatsel external
56 seqfeatsel C Code
Figure A.1: Flowchart of the seqfeatsel external
References
[1] O. Gillet and G. Richard, “Transcription and separation of drum signals from polyphonicmusic,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, pp. 529–540, March 2008.
[2] M. Miron, M. E. P. Davies, and F. Gouyon, “An open-source drum transcription system forpure data and max msp,” in 2013 IEEE International Conference on Acoustics, Speech andSignal Processing, pp. 221–225, May 2013.
[3] M. Zadel and G. Scavone, “Laptop performance: Techniques, tools, and a new interface de-sign,” in Proceedings of the International Computer Music Conference, pp. 643–648, 2006.
[4] E. Sinyor, C. Mckay, R. Fiebrink, D. Mcennis, and I. Fujinaga, “BEATBOX CLASSIFICA-TION USING ACE,” in Proceedings of the International Conference on Music InformationRetrieval, pp. 672–675, 2005.
[5] M. Atherton, “RHYTHM-SPEAK : MNEMONIC , LANGUAGE PLAY OR SONG ?,” inProceedings of the Inaugural International Conference on Music Communication Science.,pp. 15–18, 2007.
[6] D. Stowell and M. D. Plumbley, “Characteristics of the beatboxing vocal style,” tech. rep.,Centre for Digital Music Department of Electronic Engineering Queen Mary, University ofLondon, 2008.
[7] A. Kapur, M. Benning, and G. Tzanetakis, “QUERY-BY-BEAT-BOXING: MUSIC RE-TRIEVAL FOR THE DJ,” in Proceedings of the International Conference on Music Infor-mation Retrieval, pp. 170–178, 2004.
[8] T. B. Holmes and T. Holmes, Electronic and experimental music: pioneers in technology andcomposition. Psychology Press, 2002.
[9] S. Sanderson, “Low profile keyboard device and system for recording and scoring music,”Dec. 13 1988. US Patent 4,790,230.
[10] M. Duignan, J. Noble, P. Barr, and R. Biddle, Metaphors for Electronic Music Production inReason and Live, pp. 111–120. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004.
[11] S. Sapir, “Gestural control of digital audio environments,” Journal of New Music Research,vol. 31, no. 2, pp. 119–129, 2002.
[12] J. S. Downie, “Music information retrieval,” Annual Review of Information Science and Tech-nology, vol. 37, no. 1, pp. 295–340, 2003.
57
58 REFERENCES
[13] M. Schedl, E. Gómez, and J. Urbano, “Music information retrieval: Recent developmentsand applications,” Foundations and Trends in Information Retrieval, vol. 8, no. 2-3, pp. 127–261, 2014.
[14] E. Benetos, S. Dixon, D. Giannoulis, H. Kirchhoff, and A. Klapuri, “Automatic musictranscription: challenges and future directions,” Journal of Intelligent Information Systems,vol. 41, no. 3, pp. 407–434, 2013.
[15] C. W. Wu and A. Lerch, “Drum transcription using partially fixed non-negative matrix factor-ization,” in 2015 23rd European Signal Processing Conference (EUSIPCO), pp. 1281–1285,Aug 2015.
[16] E. Benetos, S. Ewert, and T. Weyde, “Automatic transcription of pitched and unpitchedsounds from polyphonic music,” in Acoustics, Speech and Signal Processing (ICASSP), 2014IEEE International Conference on, pp. 3107–3111, IEEE, 2014.
[17] D. P. W. Ellis, “Beat tracking by dynamic programming,” Journal of New Music Research,vol. 36, no. 1, pp. 51–60, 2007.
[18] M. E. P. Davies and M. D. Plumbley, “Context-dependent beat tracking of musical audio,”IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, pp. 1009–1020,March 2007.
[19] M. F. McKinney, D. Moelants, M. E. Davies, and A. Klapuri, “Evaluation of audio beattracking and music tempo extraction algorithms,” Journal of New Music Research, vol. 36,no. 1, pp. 1–16, 2007.
[20] J. Foote, “Visualizing music and audio using self-similarity,” in Proceedings of the seventhACM international conference on Multimedia (Part 1), pp. 77–80, ACM, 1999.
[21] C. Dittmar and D. Gärtner, “Real-time transcription and separation of drum recordings basedon NMF decomposition,” in Proc. of the Intl. Conference on Digital Audio Effects (DAFx),pp. 187–194, 2014.
[22] O. Gillet and G. Richard, “Automatic transcription of drum loops,” in 2004 IEEE Interna-tional Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp. iv–269–iv–272vol.4, May 2004.
[23] O. Gillet and G. Richard, “Drum track transcription of polyphonic music using noise sub-space projection,” in In Proc. of ISMIR, pp. 92–99, 2005.
[24] K. Tanghe, S. Degroeve, and B. D. Baets, “An algorithm for detecting and labeling drumevents in polyphonic music,” in In Proc. of First Annual Music Information Retrieval Evalu-ation eXchange, pp. 11–15, 2005.
[25] A. Roebel, J. Pons, M. Liuni, and M. Lagrangey, “On automatic drum transcription usingnon-negative matrix deconvolution and itakura saito divergence,” in 2015 IEEE InternationalConference on Acoustics, Speech and Signal Processing (ICASSP), pp. 414–418, April 2015.
[26] D. Stowell and M. D. Plumbley, “Delayed decision-making in real-time beatbox percussionclassification,” Journal of New Music Research, vol. 39, pp. 203–213, Sep 2010.
REFERENCES 59
[27] A. Hazan, “Towards automatic transcription of expressive oral percussive performances,” inProceedings of the 10th International Conference on Intelligent User Interfaces, IUI ’05,(New York, NY, USA), pp. 296–298, ACM, 2005.
[28] D. Christensen, E. R. Høeg, R. B. Lind, S. A. Nilsson, D. M. Smed, C. Sørensen, and S. P.Vinkel, “Automatic transcription of beatboxing,” tech. rep., Aalborg University, 2014.
[29] C. Mckay, R. Fiebrink, D. Mcennis, B. Li, and I. Fujinaga, “Ace: A framework for optimizingmusic classification,” in Proceedings of the International Conference on Music InformationRetrieval, pp. 42–49, 2005.
[30] T. Nakano, J. Ogata, M. Goto, and Y. Hiraga, “A drum pattern retrieval method by voicepercussion,” in In Proc. of ISMIR 2004, pp. 550–553, 2004.
[31] O. Gillet and G. Richard, “Drum loops retrieval from spoken queries,” Journal of IntelligentInformation Systems, vol. 24, no. 2, pp. 159–177, 2005.
[32] K. Hipke, M. Toomim, R. Fiebrink, and J. Fogarty, “Beatbox: end-user interactive definitionand training of recognizers for percussive vocalizations,” in Proceedings of the 2014 Inter-national Working Conference on Advanced Visual Interfaces, pp. 121–124, ACM, 2014.
[33] D. DeSantis, I. Gallagher, K. Haywood, R. Knudsen, G. Behles, J. Rang, R. Henke, andT. Slama, Ableton Reference Manual Version 9. Ableton, 2016.
[34] P. Brossier, “Man page of aubioonset,” Jan 2017.
[35] M. Malt and E. Jourdan, “Zsa. descriptors: a library for real-time descriptors analysis,” inProceedings of 5th Sound and Music Computing Conference Berlin, 2008.
[36] J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies, and M. B. Sandler, “A tutorialon onset detection in music signals,” IEEE Transactions on Speech and Audio Processing,vol. 13, pp. 1035–1047, Sept 2005.
[37] P. Masri, Computer modelling of sound for transformation and synthesis of musical signals.PhD thesis, University of Bristol, 1996.
[38] J. M. Grey and J. W. Gordon, “Perceptual effects of spectral modifications on musical tim-bres,” The Journal of the Acoustical Society of America, vol. 63, no. 5, pp. 1493–1500, 1978.
[39] G. Peeters, “A large set of audio features for sound description (similarity and classification)in the CUIDADO project,” tech. rep., IRCAM, 2004.
[40] D. Giannoulis, M. Massberg, and J. D. Reiss, “Parameter automation in a dynamic rangecompressor,” Journal of the Audio Engineering Society, vol. 61, no. 10, pp. 716–726, 2013.
[41] S. Dubnov, “Generalization of spectral flatness measure for non-gaussian linear processes,”IEEE Signal Processing Letters, vol. 11, pp. 698–701, Aug 2004.
[42] T. Gulzar, A. Singh, and S. Sharma, “Comparative analysis of lpcc, mfcc and bfcc for therecognition of hindi words using artificial neural networks,” International Journal of Com-puter Applications, vol. 101, no. 12, pp. 22–27, 2014.
[43] A. W. Whitney, “A direct method of nonparametric measurement selection,” IEEE Trans.Comput., vol. 20, pp. 1100–1103, Sept. 1971.
60 REFERENCES
[44] L. E. Peterson, “K-nearest neighbor,” Scholarpedia, vol. 4, no. 2, p. 1883, 2009. revision#136646.
[45] P. Brossier, MartinHN, N. Philippsen, T. Seaver, E. Müller, and S. Alexander, “aubio/aubio:0.4.5.” https://doi.org/10.5281/zenodo.496134. Accessed: 2017-5-17.
[46] W. Brent, “A timbre analysis and classification toolkit for pure data,” in ICMC, 2010.
[47] “Max API.” https://cycling74.com/sdk/MaxSDK-7.1.0/html/index.html.Accessed: 2017-5-20.