JOINT STRUCTURE ANALYSIS WITH APPLICATIONS TO MUSICANNOTATION AND SYNCHRONIZATION

Meinard M ullerSaarland University and MPI Informatik

Campus E1 4, 66123 Saarbrucken, Germanymeinard@mpi-inf.mpg.de

Sebastian EwertBonn University, Computer Science IIIRomerstr. 164, 53117 Bonn, Germanyewerts@iai.uni-bonn.de

ABSTRACT

The general goal of music synchronization is to automati-cally align different versions and interpretations related toa given musical work. In computing such alignments, re-cent approaches assume that the versions to be aligned cor-respond to each other with respect to their overall globalstructure. However, in real-world scenarios, this assump-tion is often violated. For example, for a popular song thereoften exist various structurally different album, radio, or ex-tended versions. Or, in classical music, different recordingsof the same piece may exhibit omissions of repetitions orsignificant differences in parts such as solo cadenzas. In thispaper, we introduce a novel approach for automatically de-tecting structural similarities and differences between twogiven versions of the same piece. The key idea is to performa single structural analysis for both versions simultaneouslyinstead of performing two separate analyses for each of thetwo versions. Such a joint structure analysis reveals the re-petitions within and across the two versions. As a furthercontribution, we show how this information can be used forderiving musically meaningful partial alignments and anno-tations in the presence of structural variations.

1 INTRODUCTION

Modern digital music collections contain an increasingnumber of relevant digital documents for a single musicalwork comprising various audio recordings, MIDI files, orsymbolic score representations. In order to coordinate themultiple information sources, various synchronization pro-cedures have been proposed to automatically align musi-cally corresponding events in different versions of a givenmusical work, see [1, 7, 8, 9, 14, 15] and the referencestherein. Most of these procedures rely on some variant ofdynamic time warping (DTW) and assume a global corre-spondence of the two versions to be aligned. In real-worldscenarios, however, different versions of the same piece mayexhibit significant structural variations. For example, inthecase of Western classical music, different recordings often

The research was funded by the German Research Foundation (DFG)and the Cluster of Excellence on Multimodal Computing and Interaction.

exhibit omissions of repetitions (e. g., in sonatas and sym-phonies) or significant differences in parts such as solo ca-denzas of concertos. Similarly, for a given popular, folk, orart song, there may be various recordings with a differentnumber of stanzas. In particular for popular songs, theremay exist structurally different album, radio, or extendedversions as well as cover versions.

A basic idea to deal with structural differences in thesynchronization context is to combine methods from mu-sic structure analysis and music alignment. In a first step,one may partition the two versions to be aligned into musi-cally meaningful segments. Here, one can use methods fromautomated structure analysis [3, 5, 10, 12, 13] to derive sim-ilarity clusters that represent the repetitive structure of thetwo versions. In a second step, the two versions can then becompared on the segment level with the objective for match-ing musically corresponding passages. Finally, each pair ofmatched segments can be synchronized using global align-ment strategies. In theory, this seems to be a straightforwardapproach. In practise, however, one has to deal with severalproblems due to the variability of the underlying data. Inparticular, the automated extraction of the repetitive struc-ture constitutes a delicate task in case the repetitions revealsignificant differences in tempo, dynamics, or instrumen-tation. Flaws in the structural analysis, however, may beaggravated in the subsequent segment-based matching stepleading to strongly corrupted synchronization results.

The key idea of this paper is to perform a single, jointstructure analysis for both versions to be aligned, which pro-vides richer and more consistent structural data than in thecase of two separate analyses. The resulting similarity clus-ters not only reveal the repetitions within and across the twoversions, but also induce musically meaningful partial align-ments between the two versions. In Sect. 2, we describe ourprocedure for a joint structure analysis. As a further con-tribution of this paper, we show how the joint structure canbe used for deriving a musically meaningful partial align-ment between two audio recordings with structural differ-ences, see Sect. 3. Furthermore, as described in Sect. 4,our procedure can be applied for automatic annotation of agiven audio recording by partially available MIDI data. InSect. 5, we conclude with a discussion of open problems and

0 100 200 3000

50

100

150

200

250

300

350

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 100 200 3000

50

100

150

200

250

300

350

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 100 200 300

1

2

1 2 3 4

1

2

20406080100120

0 100 200 300

1

2

0 100 2000

50

100

0

50

100

150

200

0 100 2000

50

100

0

0.5

1

(a) (b)

(c)

(d)

(e)

(f)

(g)

A1

B1 A

2

1 A2

2 B2

1 B2

2

A1

B1

A2

1

A2

2

B2

1

B2

2

A1

B1 A

2

1 A2

2 B2

1 B2

2 A1

B1 A

2

1 A2

2 B2

1 B2

2

Figure 1. Joint structure analysis and partial synchronization for two structurally different versions of the Aria of the Goldberg VariationsBWV 988 by J.S. Bach. The first version is played by G. Gould (musical form A

1B

1) and the second by M. Perahia (musical formA

2

1A2

2B2

1B2

2 ). (a) Joint similarity matrixS . (b) Enhanced matrix and extracted paths.(c) Similarity clusters. (d) Segment-based scorematrixM and match (black dots).(e) Matched segments.(f) Matrix representation of matched segments.(g) Partial synchronization result.

prospects on future work.The problem of automated partial music synchronization

has been introduced in [11], where the idea is to use the con-cept of path-constrained similarity matrices to enforce mu-sically meaningful partial alignments. Our approach carriesthis idea even further by using cluster-constraint similaritymatrices, thus enforcing structurally meaning partial align-ments. A discussion of further references is given in thesubsequent sections.

2 JOINT STRUCTURE ANALYSIS

The objective of a joint structure analysis is to extract there-petitive structure within and across two different music rep-resentations referring to the same piece of music. Each ofthe two versions can be an audio recording, a MIDI version,or a MusicXML file. The basic idea of how to couple thestructure analysis of two versions is very simple. First, oneconverts both versions into common feature representationsand concatenates the resulting feature sequences to form asingle long feature sequence. Then, one performs a com-mon structure analysis based on the long concatenated fea-ture sequence. To make this strategy work, however, onehas to deal with various problems. First, note that basi-cally all available procedures for automated structure analy-sis have a computational complexity that is at least quadraticin the input length. Therefore, efficiency issues becomecrucial when considering a single concatenated feature se-quence. Second, note that two different versions of thesame piece often reveal significant local and global tempodifferences. Recent approaches to structure analysis suchas [5, 12, 13], however, are built upon the constant tempoassumption and cannot be used for a joint structure analysis.Allowing also tempo variations between repeating segmentsmakes the structure analysis problem a much harder prob-lem [3, 10]. We now summarize the approach used in thispaper closely following [10].

Given two music representations, we transform them

into suitable feature sequencesU := (u1, u2, . . . , uL) andV := (v1, v2, . . . , vM ), respectively. To reduce differenttypes of music data (audio, MIDI, MusicXML) to the sametype of representation and to cope with musical variationsin instrumentation and articulation, chroma-based featureshave turned out to be a powerful mid-level music repre-sentation [2, 3, 8]. In the subsequent discussion, we em-ploy a smoothed normalized variant of chroma-based fea-tures (CENS features) with a temporal resolution of1 Hz,see [8] for details. In this case, each12-dimensional featurevectoruℓ, ℓ ∈ [1 : L], andvm, m ∈ [1 : M ], expressesthe local energy of the audio (or MIDI) distribution in the12 chroma classes. The feature sequences strongly correlateto the short-time harmonic content of the underlying musicrepresentations. We now define the sequenceW of lengthN := L + M by concatenating the sequencesU andV :

W := (w1, w2, . . . , wN ) := (u1, . . . , uL, v1, . . . , vM ).

Fixing a suitable local similarity measure— here, we use theinner vector product—the(N×N)-joint similarity matrix Sis defined byS(i, j) := 〈wi, wj〉, i, j ∈ [1 : N ]. Each tuple(i, j) is called acell of the matrix. Apath is a sequencep =(p1, . . . , pK) with pk = (ik, jk) ∈ [1 : N ]2, k ∈ [1 : K],satisfying1 ≤ i1 ≤ i2 ≤ . . . ≤ iK ≤ N and1 ≤ j1 ≤j2 ≤ . . . ≤ jK ≤ N (monotonicity condition) as well aspk+1 − pk ∈ Σ, whereΣ denotes a set of admissible stepsizes. In the following, we useΣ = {(1, 1), (1, 2), (2, 1)}.

As an illustrative example, we consider two differentaudio recordings of the Aria of the Goldberg VariationsBWV 988 by J.S. Bach, in the following referred to asBachexample. The first version with a duration of115 seconds isplayed by Glen Gould without repetitions (corresponding tothe musical formA1B1) and the second version with a du-ration of241 seconds is played by Murray Perahia with re-petitions (corresponding to the musical formA2

1A22B

21B2

2 ).For the feature sequences holdL = 115, M = 241, andN = 356. The resulting joint similarity matrix is shown in

0 100 200 300 400 500 6000

100

200

300

400

500

600

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 100 200 300 400 500 600

5

10

15

10 20 30

5

10

15

20

25

30

0

20

40

60

80

100

0 100 200 300 4000

50

100

150

200

250

0

100

200

300

400

500

0 100 200 300 4000

50

100

150

200

250

0

0.2

0.4

0.6

0.8

1

(a) (b)

(c)

(d)

(e)

Figure 2. Joint structure analysis and partial synchronization for two structurally modified versions of Beethoven’s Fifth Symphony Op. 67.The first version is a MIDI version and the second one an audio recording by Bernstein.(a) Enhanced joint similarity matrix and extractedpaths.(b) Similarity clusters.(c) Segment-based score matrixM and match (indicated by black dots).(d) Matrix representation of matchedsegments.(e) Partial synchronization result.

Fig. 1a, where the boundaries between the two versions areindicated by white horizontal and vertical lines.

In the next step, the path structure is extracted from thejoint similarity matrix. Here, the general principle is thateach path of low cost running in a direction along the maindiagonal (gradient(1, 1)) corresponds to a pair of similarfeature subsequences. Note that relative tempo differencesin similar segments are encoded by the gradient of the path(which is then in a neighborhood of(1, 1)). To ease thepath extraction step, we enhance the path structure ofS bya suitable smoothing technique that respects relative tempodifferences. The paths can then be extracted by a robust andefficient greedy strategy, see Fig. 1b. Here, because of thesymmetry ofS, one only has to consider the upper left partof S. Furthermore, we prohibit paths crossing the bound-aries between the two versions. As a result, each extractedpath encodes a pair of musically similar segments, whereeach segment entirely belongs either to the first or to the sec-ond version. To determine the global repetitive structure,weuse a one-step transitivity clustering procedure, which bal-ances out the inconsistencies introduced by inaccurate andincorrect path extractions. For details, we refer to [8, 10].

Altogether, we obtain a set of similarity clusters. Eachsimilarity cluster in turn consists of a set of pairwise similarsegments encoding the repetitions of a segment within andacross the two versions. Fig. 1c shows the resulting set ofsimilarity clusters for our Bach example. Both of the clus-ters consist of three segments, where the first cluster corre-sponds to the threeB-partsB1, B2

1 , andB22 and the sec-

ond cluster to the threeA-partsA1, A21, andA2

2. The jointanalysis has several advantages compared to two separateanalyses. First note that, since there are no repetitions inthefirst version, a separate structure analysis for the first version

would not have yielded any structural information. Second,the similarity clusters of the joint structure analysis naturallyinduce musically meaningful partial alignments between thetwo versions. For example, the first cluster shows thatB1

may be aligned toB21 or to B2

2 . Finally, note that the deli-cate path extraction step often results in inaccurate and frag-mented paths. Because of the transitivity step, the joint clus-tering procedure balances out these flaws and compensatesfor missing parts to some extent by using joint informationacross the two versions.

On the downside, a joint structural analysis is computa-tionally more expensive than two separate analyses. There-fore, in the structure analysis step, our strategy is to use arelatively low feature resolution of1 Hz. This resolutionmay then be increased in the subsequent synchronizationstep (Sect. 3) and annotation application (Sect. 4). Our cur-rent MATLAB implementation can easily deal with an over-all length up toN = 3000 corresponding to more then fortyminutes of music material. (In this case, the overall com-putation time adds up to10-400 seconds with the path ex-traction step being the bottleneck, see [10]). Thus, our im-plementation allows for a joint analysis even for long sym-phonic movements of a duration of more than20 minutes.

Another drawback of the joint analysis is that local in-consistencies across the two versions may cause an over-fragmentation of the music material. This may result ina large number of incomplete similarity clusters contain-ing many short segments. As an example, we consider aMIDI version as well as a Bernstein audio recording of thefirst movement of Beethoven’s Fifth Symphony Op. 67. Westructurally modified both versions by removing some sec-tions. Fig. 2a shows the enhanced joint similarity matrixand Fig. 2b the set of joint similarity clusters. Note that

some of the resulting16 clusters contain semantically mean-ingless segments stemming from spuriously extracted pathfragments. At this point, one could try to improve the over-all structure result by a suitable postprocessing procedure.This itself constitutes a difficult research problem and is notin the scope of this paper. Instead, we introduce a proce-dure for partial music alignment, which has some degree ofrobustness to inaccuracies and flaws in the previously ex-tracted structural data.

3 PARTIAL SYNCHRONIZATION

Given two different representations of the same underlyingpiece of music, the objective ofmusic synchronization is toautomatically identify and link semantically correspondingevents within the two versions. Most of the recent synchro-nization approaches use some variant of dynamic time warp-ing (DTW) to align the feature sequences extracted fromthe two versions, see [8]. In classical DTW, all elementsof one sequence are matched to elements in the other se-quence (while respecting the temporal order). This is prob-lematic when elements in one sequence do not have suit-able counterparts in the other sequence. In the presence ofstructural differences between the two sequences, this typ-ically leads to corrupted and musically meaningless align-ments [11]. Also more flexible alignment strategies such assubsequence DTW or partial matching strategies as used inbiological sequence analysis [4] do not properly account forsuch structural differences.

A first approach for partial music synchronization hasbeen described in [11]. Here, the idea is to first constructa path-constrained similarity matrix, which a priori con-stricts possible alignment paths to a semantically meaning-ful choice of admissible cells. Then, in a second step, apath-constrained alignment can be computed using standardmatching procedures based on dynamic programming.

We now carry this idea even further by using the seg-ments of the joint similarity clusters as constraining ele-ments in the alignment step. To this end, we consider pairsof segments, where the two segments lie within the samesimilarity cluster and belong to different versions. Moreprecisely, letC = {C1, . . . , CM} be the set of clusters ob-tained from the joint structure analysis. Each similarity clus-ter Cm, m ∈ [1 : M ], consists of a set of segments (i. e.,subsequences of the concatenated feature sequenceW ). Letα ∈ Cm be such a segment. Then letℓ(α) denote thelength ofα and c(α) := m the cluster affiliation. Recallthatα either belongs to the first version (i. e.,α is a subse-quence ofU ) or to the second version (i. e.,α is a subse-quence ofV ). We now form two lists of segments. The firstlist (α1, . . . , αI) consists of all those segments that are con-tained in some cluster ofC and belong to the first version.The second list(β1, . . . , βJ) is defined similarly, where thesegments now belong to the second version. Both lists are

sorted according to the start positions of the segments. (Incase two segments have the same start position, we breakthe tie by also considering the cluster affiliation.) We definea segment-basedI × J-score matrixM by

M(i, j) :=

{

ℓ(αi) + ℓ(βj) for c(αi) = c(βj),0 otherwise,

i ∈ [1 : I], j ∈ [1 : J ]. In other words,M(i, j) is positiveif and only if αi andβj belong to the same similarity clus-ter. Furthermore,M(i, j) depends on the lengths of the twosegments. Here, the idea is to favor long segments in thesynchronization step. For an illustration, we consider theBach example of Fig. 1, where(α1, . . . , αI) = (A1, B1)and(β1, . . . , βJ ) = (A2

1, A22, B

21 , B2

2). The resulting matrixM is shown in Fig. 1d. For another more complex example,we refer to Fig. 2c.

Now, a segment-basedmatch is a sequenceµ =(µ1, . . . , µK) with µk = (ik, jk) ∈ [1 : I] × [1 : J ] fork ∈ [1 : K] satisfying1 ≤ i1 < i2 < . . . < iK ≤ I

and1 ≤ j1 < j2 < . . . < jK ≤ J . Note that a matchinduces a partial assignment of segment pairs, where eachsegment is assigned to at most one other segment. Thescore of a matchµ with respect toM is then defined as∑K

k=1M(ik, jk). One can now use standard techniques to

compute a score-maximizing match based on dynamic pro-gramming, see [4, 8]. For details, we refer to the litera-ture. In the Bach example, the score-maximizing matchµ isgiven byµ = ((1, 1), (2, 3)). In other words, the segmentα1 = A1 of the first version is assigned to segmentβ1 = A2

1

of the second version andα2 = B1 is assigned toβ3 = B21 .

In principle, the score-maximizing matchµ constitutesour partial music synchronization result. To make the pro-cedure more robust to inaccuracies and to remove clusterredundancies, we further clean the synchronization resultina postprocessing step. To this end, we convert the score-maximizing matchµ into a sparsepath-constrained simi-larity matrix Spath of size L × M , whereL and M arethe lengths of the two feature sequencesU and V , re-spectively. For each pair of matched segments, we com-pute an alignment path using a global synchronization algo-rithm [9].Each cell of such a path defines a non-zero entryof Spath, where the entry is set to the length of the path (thusfavoring long segments in the subsequent matching step).All other entries of the matrixSpath are set to zero. Fig. 1fand Fig. 2d show the resulting path-constrained similaritymatrices for the Bach and Beethoven example, respectively.Finally, we apply the procedure as described in [11] us-ing Spath (which is generally much sparser than the path-constrained similarity matrices as used in [11])to obtain apurified synchronization result, see Fig. 1g and Fig. 2e.

To evaluate our synchronization procedure, we per-formed similar experiments as described in [11]. In oneexperiment, we formed synchronization pairs each consist-ing of two different versions of the same piece. Each pair

(a)

0 100 200 300 400 5000

100

200

300

400

0 100 200 300 400 500D

C

B

AS1 S2 S3 S4 S5 S6

P1 P2 P3

(b)

0 50 100 150 200 2500

50

100

150

0 50 100 150 200 250D

C

B

AS1 S2 S3 S4

P1 P2

(c)

0 50 100 150 200 2500

100

200

300

0 50 100 150 200 250D

C

B

AS1 S2 S3 S4 S5 S6 S7

P1 P2 P3

(d)

0 50 1000

20

40

60

80

0 50 100D

C

B

AS1 S2 S3 S4 S5 S6 S7

P1

Figure 3. Partial synchronization results for various MIDI-audio synchronization pairs. The top figures show the final path componentsof the partial alignments and the bottom figures indicate theground truth (Row A), the final annotations (Row B), and a classification intocorrect (Row D) and incorrect annotations (Row C), see text for additional explanations. The pieces are specified in Table 1. (a) Haydn (RWCC001),(b) Schubert (RWC C048, distorted),(c) Burke (P093),(d) Beatles (“Help!”, distorted).

consists either of an audio recording and a MIDI version orof two different audio recordings (interpreted by differentmusicians possibly in different instrumentations). We man-ually labeled musically meaningful sections of all versionsand then modified the pairs by randomly removing or dupli-cating some of the labeled sections, see Fig. 3. The partialsynchronization result computed by our algorithm was ana-lyzed by means of its path components. A path componentis said to becorrect if it aligns corresponding musical sec-tions. Similarly, a match is said to becorrect if it covers(up to a certain tolerance) all semantically meaningful cor-respondences between the two versions (this information isgiven by the ground truth) and if all its path components arecorrect. We tested our algorithm on more than387 differentsynchronization pairs resulting in a total number of1080path components. As a result,89% of all path componentsand71% of all matches were correct (using a tolerance of3seconds).

The results obtained by our implementation of thesegment-based synchronization approach are qualitativelysimilar to those reported in [11]. However, there is one cru-cial difference in the two approaches. In [11], the authorsuse a combination of various ad-hoc criteria to constructa path-constrained similarity matrix as basis for their par-tial synchronization. In contrast, our approach uses only thestructural information in form of the joint similarity clustersto derive the partial alignment. Furthermore, the availabil-ity of structural information within and across the two ver-sions allows for recovering missing relations based on suit-able transitivity considerations. Thus, each improvementofthe structure analysis will have a direct positive effect onthequality of the synchronization result.

4 AUDIO ANNOTATION

The synchronization of an audio recording and a corre-sponding MIDI version can be regarded as an automatedannotation of the audio recording by means of the explicitnote events given by the MIDI file. Often, MIDI versionsare used as a kind of score-like symbolic representation of

the underlying musical work, where redundant informationsuch as repetitions are not encoded explicitly. This is a fur-ther setting with practical relevance where two versions tobe aligned have a different repetitive structure (an audio ver-sion with repetitions and a score-like MIDI version withoutrepetitions). In this setting, one can use our segment-basedpartial synchronization to still obtain musically adequate au-dio annotations.

We now summarize one of our experiments, which hasbeen conducted on the basis of synchronization pairs con-sisting of structurally equivalent audio and MIDI versions. 1

We first globally aligned the corresponding audio and MIDIversions using a temporally refined version of the synchro-nization procedure described in [9]. These alignments weretaken as ground truth for the audio annotation. Similar tothe experiment of Sect. 3, we manually labeled musicallymeaningful sections of the MIDI versions and randomly re-moved or duplicated some of these sections. Fig. 3a il-lustrates this process by means of the first movement ofHaydn’s Symphony No. 94 (RWC C001).Row A of thebottom part shows the original six labeled sections S1 toS6 (warped according to the audio version). In the modi-fication, S2 was removed (no line) and S4 was duplicated(thick line). Next, we partially aligned the modified MIDIwith the original audio recording as described in Sect. 3.The resulting three path components of our Haydn exampleare shown in the top part of Fig. 3a. Here, the vertical axiscorresponds to the MIDI version and the horizontal axis tothe audio version. Furthermore,Row B of the bottom partshows the projections of the three path components onto theaudio axis resulting in the three segments P1, P2, and P3.These segments are aligned to segments in the MIDI thusbeing annotated by the corresponding MIDI events. Next,we compared these partial annotations with the ground truthannotations on the MIDI note event level. We say that analignment of a note event to a physical time position of theaudio version is correct in aweak (strong) sense, if there is

1 Most of the audio and MIDI files were taken from the RWC musicdatabase [6]. Note that for the classical pieces, the original RWC MIDIand RWC audio versions are not aligned.

Original DistortedComposer Piece RWCweak strong weak strong

Haydn Symph. No. 94, 1st Mov.C001 98 97 97 95Beethoven Symph. Op. 67, 1st Mov.C003 99 98 95 91Beethoven Sonata Op. 57, 1st Mov. C028 99 99 98 96Chopin Etude Op. 10, No. 3 C031 93 93 93 92Schubert Op. 89, No. 5 C048 97 96 95 95Burke Sweet Dreams P093 88 79 74 63Beatles Help! — 97 96 77 74

Average 96 94 91 87

Table 1. Examples for automated MIDI-audio annotation (mostof files are from the RWC music database [6]). The columns showthe composer, the piece of music, the RWC identifier, as well as theannotation rate (in%) with respect to the weak and strong criterionfor the original MIDI and some distorted MIDI.

a ground truth alignment of a note event of the same pitch(and, in the strong case, additionally lies in the same musicalcontext by checking an entire neighborhood of MIDI notes)within a temporal tolerance of100 ms. In our Haydn exam-ple, the weakly correct partial annotations are indicated inRow D and the incorrect annotations inRow C.

The other examples shown in Fig. 3 give a representa-tive impression of the overall annotation quality. Generally,the annotations are accurate—only at the segment bound-aries there are some larger deviations. This is due to ourpath extraction procedure, which often results in “frayed”path endings. Here, one may improve the results by cor-recting the musical segment boundaries in a postprocessingstep based on cues such as changes in timbre or dynamics.A more critical example (Beatles example) is shown Fig. 3d,where we removed two sections (S2 and S7) from the MIDIfile and temporally distorted the remaining parts. In this ex-ample, the MIDI and audio version also exhibit significantdifferences on the feature level. As a result, an entire sec-tion (S1) has been left unannotated leading to a relativelypoor rate of77% (74%) of correctly annotated note eventswith respect to the weak (strong) criterion.

Finally, Table 1 shows further rates of correctly anno-tated note events for some representative examples. Ad-ditionally, we have repeated our experiments with sig-nificantly temporally distorted MIDI files (locally up to±20%). Note that most rates only slightly decrease (e. g.,for the Schubert piece, from97% to 95% with respectto the weak criterion), which indicates the robustness ofour overall annotation procedure to local tempo differ-ences. Further results as well as audio files of sonificationscan be found athttp://www-mmdb.iai.uni-bonn.de/projects/partialSync/

5 CONCLUSIONS

In this paper, we have introduced the strategy of performingajoint structural analysis to detect the repetitive structure within andacross different versions of the same musical work. As a corecomponent for realizing this concept, we have discussed a struc-ture analysis procedure that can cope with relative tempo differ-ences between repeating segments. As further contributions, we

have shown how joint structural information can be used to dealwith structural variations in synchronization and annotation appli-cations. The tasks ofpartial music synchronization and annotationis a much harder then theglobal variants of these tasks. The rea-son for this is that in the partial case one needsabsolute similaritycriteria, whereas in the global case one only requiresrelative cri-teria. One main message of this paper is that automated musicstructure analysis is closely related to partial music alignment andannotation applications. Hence, improvements and extensions ofcurrent structure analysis procedures to deal with variouskinds ofvariations is of fundamental importance for future research.

6 REFERENCES

[1] V. Arifi, M. Clausen, F. Kurth, and M. Muller. Synchronizationof music data in score-, MIDI- and PCM-format.Computing inMusicology, 13, 2004.

[2] M.A. Bartsch, G.H. Wakefield: Audio thumbnailing of popu-lar music using chroma-based representations. IEEE Trans.onMultimedia7(1) (2005) 96–104.

[3] R. Dannenberg, N. Hu,Pattern discovery techniques for musicaudio, Proc. ISMIR, Paris, France, 2002.

[4] R. Durbin, S. Eddy, A. Krogh, and G. Mitchison,BiologicalSequence Analysis : Probabilistic Models of Proteins and Nu-cleic Acids, Cambridge Univ. Press, 1999.

[5] M. Goto, A chorus section detection method for musical au-dio signals and its application to a music listening station,IEEE Transactions on Audio, Speech & Language Processing14 (2006), no. 5, 1783–1794.

[6] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka. RWC mu-sic database: Popular, classical and jazz music databases.Proc.ISMIR, Paris, France, 2002.

[7] N. Hu, R. Dannenberg, and G. Tzanetakis. Polyphonic audiomatching and alignment for music retrieval. InProc. IEEEWASPAA, New Paltz, NY, October 2003.

[8] M. Muller: Information Retrieval for Music and Motion.Springer (2007).

[9] M. Muller, H. Mattes, and F. Kurth. An efficient multiscaleapproach to audio synchronization. InProc. ISMIR, Victoria,Canada, pages 192–197, 2006.

[10] M. Muller, F. Kurth, Towards structural analysis of audiorecordings in the presence of musical variations, EURASIPJournal on Advances in Signal Processing 2007, Article ID89686, 18 pages.

[11] M. Muller, D. Appelt: Path-constrained partial musicsyn-chronization. In: Proc. International Conference on Acoustics,Speech, and Signal Processing, Las Vegas, USA (2008).

[12] G. Peeters,Sequence representation of music structure usinghigher-order similarity matrix and maximum-likelihood ap-proach, Proc. ISMIR, Vienna, Austria, 2007.

[13] C. Rhodes, M. Casey,Algorithms for determining and la-belling approximate hierarchical self-similarity, Proc. ISMIR,Vienna, Austria, 2007.

[14] F. Soulez, X. Rodet, and D. Schwarz. Improving polyphonicand poly-instrumental music to score alignment. InProc. IS-MIR, Baltimore, USA, 2003.

[15] R. J. Turetsky and D. P. Ellis. Force-Aligning MIDI Synthe-ses for Polyphonic Music Transcription Generation. InProc.ISMIR, Baltimore, USA, 2003.