A Musical Structure Model and Its Visual Mechanism for

Hypertext Score Systems

Yusuke Ariyoshi,1,*

Shinji Shimojo,2 and Hideo Miyahara

1

1Faculty of Engineering Science, Osaka University, Toyonaka, 560-8531 Japan

2Computation Center, Osaka University, Ibaraki, 567-0047 Japan

SUMMARY

In order to perform with consistency a musical score

having multiple parts, as in an orchestra or a chorus, it is

important that the conductor and performers share its mu-

sical semantic structures. However, past studies of the ex-

pression or visualization of musical semantic structure in

the field of music information processing have dealt with

scores having a single voice part. In this paper, we propose

a new electronic musical score system, �HyperScore,�

based on the hypertext data model. Expanding the hypertext

data model and its visualization mechanism, HyperScore

has a flexible semantic structure model and a strong visu-

alization mechanism which help the conductor and per-

formers understand the semantics of the score and

communicate with each other. The musical semantic struc-

ture model consists of three components: phrase�motif�

note tree structures of the musical score, musical structures

such as particular musical forms, and links among these

structures. The visualization mechanism generates various

views of the semantic structure of the music from different

perspectives. The view is generated by a visualization

mechanism specifying four parameters: the XY axes, dis-

play rules, selection, and granularity. The view illustrates

the musical semantics of series of notes clearly by their

position in the view and their display attributes. © 2001

Scripta Technica, Syst Comp Jpn, 32(7): 39�51, 2001

Key words: Hypertext; musical score system; mu-

sical semantic structure; hypermedia; information visuali-

zation.

1. Introduction

Many people consider that the process of creating a

performance consists of practicing the score of a musical

piece. However, in reality, the performance, that is, the plan

for the execution of the various parts piece from the score,

exists before practice begins. Thus, practice and perform-

ance are based on this performance plan. In addition, the

performance plan is refined by using the results of practice

and performance (Fig. 1).

© 2001 Scripta Technica

Systems and Computers in Japan, Vol. 32, No. 7, 2001Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J81-D-II, No. 9, September 1998, pp. 1937�1948

*Currently affiliated with NEC Human Media Research Laboratories Fig. 1. Process of making performance.

39

The structure of a musical piece, which is the basis

of its performance plan, is a complicated interweaving of

multiple elements such as musical form, rhythmic structure,

and the like. In addition, the structure of the piece is not

clearly indicated in the score and ambiguity is associated

with differences in interpretation of the score. Although the

structure of the piece is usually written into the score, there

are limits to such written expression due to the complexity

and ambiguity noted above.

In a musical piece with multiple parts, such as a piece

for an orchestra or chorus, which is the object of study of

this paper, performance creation involves a division of labor

between the conductor and the performers. First, the con-

ductor creates a performance plan for the entire piece

according to his interpretation of it. Next, the performers of

the parts create detailed performance plans of their parts

based on the musical structure and the conductor�s perform-

ance plan, and the rehearsals and performances are con-

ducted in accordance with these plans.

The conductor needs to impart his knowledge or

interpretation of the musical structure of a piece with mul-

tiple parts to the performers; a performance without consis-

tency results when the musical structure of the piece is not

imparted well, such as when expressions differ between the

voice parts or there is no unity between the conductor and

the performers. On the other hand, the musical structure is

more complicated because of the multiple parts. Thus, the

limitations on expression of the musical structure in a score

with multiple parts are more serious.

In this paper, in order to give support to performance

creation, we propose a musical structure model expressing

a complicated musical structure of a piece and a musical

score hypertext system, called HyperScore, which has a

mechanism for visualizing the musical structure as in the

score.

The musical structure model of HyperScore is an

expanded structured hypertext model based on methods of

analysis of musical pieces, and expresses three elements of

the score structure, namely, musical elements, the logical

structure of the piece, and attribute links. The mechanism

for visualization of the musical piece structure generates a

view display of a complicated musical piece seen from

various points of view by specifying four parameters�se-

lection, granularity, XY axes, and display rules�based on

analyses of the interpretations of the musical piece structure

by the conductor and the performers.

2. Musical Piece Structure Model

In order to support the creation of performance plans,

which is a goal of HyperScore, it is important to express

simply and concisely the parts of the musical piece structure

related to the creation of performance plans. Thus, the

requirements for achieving the above task are analyzed and

then a musical piece structure for flexibly expressing the

structure of the piece is proposed.

2.1. Analyses of musical piece structure

The musical structure, including elements such as the

musical form, harmonic structure, and so on, is extracted

by musical analysis (the entire logical structure of the piece

is called the overall musical structure and is differentiated

from individual musical structures). This musical structure

gives musicological meaning to sequences of notes,

phrases, and the like (Fig. 2). Thus, the performance plan

for the sequence of notes is decided by this musicological

meaning. [1].

The authors interviewed conductors and performers

and surveyed the methods of analyzing a musical piece.

Based on the results, we found that the structures governing

the performance plan are of four kinds: giving musical

meaning to a sequence of notes from musical structures

recognized from the past; giving musical meaning in terms

of elementary musical structures; inclusion relationships of

musical piece elements; and giving relationships by musi-

cal structures.

(1) Giving musical meaning to a sequence of notes

based on the musical structure

For example, in determining a performance plan of a

certain phrase, the performance plan �this soprano phrase

is to be performed so that it clearly stands out, since it is the

beginning of the recurrence of the first theme (musical

meaning)� might be selected.

The performance plan thus is determined by the

structure through which the musical structure gives musical

meaning to a sequence of notes. For example, the type of

musical structure called a musical form directly imparts to

Fig. 2. Analysis of score.

40

a phrase the musical meaning of �recurrence of the first

theme.�

Although the performance plan in this example is not

a detailed plan but merely a plan to �perform so that it

clearly stands out,� in reality, it is incorporated into a more

detailed performance plan by the fact that a musical mean-

ing different from a multiple number of musical structures

is given to one sequence of notes.

(2) Giving meaning to a musical structure in terms of

elementary musical structures

Musical pieces having multiple parts, which are the

targets of HyperScore, frequently have very complicated

musical structures. For example, a symphony can have

several dozen voice parts and more than 100 phrases. Since

the more difficult the performance plan that can be derived,

the larger the number of musical structures, many musical

structures are analyzed in groups and multistage analyses

are performed to create performance plans (simple musical

structures which are results of structural analysis are re-

ferred to below as elemental musical structures).

Taking a simple example, Fig. 3 shows that a musical

structure with a compound ternary form can be decomposed

into elemental musical structures of ternary and binary

form. The performance plan chosen for phrase C1 is one

that combines requirements derived from the musical

meanings of both the third block of the ternary form and the

first block of the binary form.

By performing multistage analyses in this way, the

creation of a performance for a large musical piece is made

simpler by deriving performance plans from the musical

meanings given by the simpler elemental musical struc-

tures, even if the piece has a large number of musical

structures for which performance plans cannot easily be

derived directly.

When multistage analyses are performed, musical

meanings are attached to the original musical structures by

the elementary musical structures, in such a way that the

performance plans can be derived from the musical mean-

ing given to a sequence of notes by musical structures.

Giving musical meaning to a musical structure in terms of

elementary musical structures thus consists of building up

meaningful structures of larger size.

(3) Inclusion relationships of note sequences

Notes, motifs, phrases, and so on, differing in granu-

larity exist in a sequence of notes having musical meaning.

There is an inclusion relationship between a note sequence

of large granularity and a note sequence of small granular-

ity, and a note sequence of large granularity is composed of

note sequences of small granularities.

The musical structures giving musical meaning differ

with the granularity of note sequences. In Fig. 2, a musical

meaning is given to a phrase in terms of three kinds of

musical structure, that is, voice part formation, musical

form, and role. On the other hand, the musical meaning of,

for example, �the third note of harmony I� or �the beginning

note of the phrase� is given at the note level, of small

granularity, in terms of the musical structure of the position

inside the harmony or the position inside the phrase.

In addition, there are cases in which the determina-

tion of a performance plan of a note sequence is influenced

by the performance plans or musical meanings of the note

sequences in the inclusion relationships.

For example, when the meaning of �the third note of

harmony I� is given to a certain note in analyses at the note

level, since the harmony is determined as major or minor

by the musical step of the third note, performing this note

at the correct pitch is required. Thus, the possibility of this

pitch being unstable is eliminated in the performance plan

of the phrase containing this note.

Specifically, a performance plan is not simply deter-

mined by the musical meanings given by the musical struc-

tures, but must be chosen to agree well with the

performance plans or musical meanings of the note se-

quences in the inclusion relationships.

Thus, the structure of the inclusion relationships of

note sequences also influences the determination of a per-

formance plan for a note sequence.

(4) Linking relationships by musical structures

For example, phrase I at the upper left is given the

musical meaning of �soprano, first phrase of A, melody�

from the musical structure of the voice part formation,

musical form, and role, and the phrase below it is given the

meanings of �alto, first phrase of A, harmony.� In addition,

the performance plan of this phrase is chosen so as to

combine the volume and tempo with the melody easily

without allowing it to stand out, since its role is harmony.Fig. 3. Music structure and element music structure.

41

In this example, a performance plan which agrees

well with the performance plan of the note sequence of the

melody, with which it has a linkage relationship in terms of

the musical structure of the role, is selected.

Specifically, the performance plan of a note sequence

is determined not only by the musical meaning of the note

sequence itself, but also by the influences of the perform-

ance plan and the musical meaning of the note sequence

with which this sequence is linked in a structural relation-

ship. Thus, in determining the performance plan of a note

sequence, the structural relationship consisting of the link-

ing between note sequences plays a role.

2.2. Musical piece structure model

The musical structure model is designed on the basis

of a structured hypertext model. The structured hypertext

model has a capability of flexible expression and has al-

ready been used to express the logical structures of docu-

ments or multimedia codings.

The basic elements constituting the hypertext model

are links, expressing the relationships between information,

and nodes, expressing the real body of information. In

structured hypertext, the links or nodes are expanded and

have the structures of a list or a tree.

A purpose of the musical piece structural model is to

express the structure determining the performance plan of

a note sequence in a musical structure. Thus, expressing the

above four kinds of structures is a requirement of the

musical structure model.

The musical structure model is constructed from four

broad elements: musical piece elements, musical piece

logic structure, attribute links, and analysis structure, and it

expresses a frame for determining the performance plan of

a note sequence by these elements.

[Musical piece elements] In the musical piece struc-

ture model, the note sequences which serve as the units in

the creation of the performance plan are expressed by links

of inclusion relationships with the musical piece element

nodes. The musical piece element nodes express note se-

quences having musical meanings, such as phrases and

motifs. The links of inclusion relationships express the

inclusion relationships of note sequences differing in

granularity. In this way the musical piece elements form a

tree structure of musical piece element nodes connected by

the links of inclusion relationships.

[Musical piece theory structure] In the musical piece

structure model, the musical structures such as the musical

form are expressed by links expressing the hierarchical

relationships between musical attribute nodes and attribute

values. The musical attribute nodes express the musical

meanings given to the note sequences, and the hierarchical

relationship links express the relationships between the

musical meanings.

For example, Fig. 5 shows the logical structure of a

musical piece composed of four mixed voice parts. The four

mixed voice parts are composed of male voice parts and

female voice parts, the male voice parts are composed of

the tenor and base parts, and the female voice parts are

composed of the soprano and alto parts. In this example,

the musical meanings of the male and female voice parts or

of the four voice parts are expressed by the musical attribute

nodes, and the male voice parts are expressed by linking the

musical attribute nodes in the relationships of the tenor and

base compositions by hierarchical relationship links. Thus,

the musical structure has a tree structure consisting of

musical attribute nodes connected by the hierarchical rela-

tionship links.

[Attribute linking] In the musical piece structure

model, the relationships giving musical meaning are ex-

pressed by attribute linking links. Figure 6 shows an exam-

ple of linking a meaning to a note sequence from the

musical structure. The attribute links are drawn from the

musical attribute nodes in the musical structures of the

musical form, the voice part formation, and the role to the

musical element nodes expressing phrases. Figure 7 is an

example of linking meanings in multistage analysis. The

attribute-linking links are drawn from the musical attribute

nodes of the elementary musical structures of binary form

and ternary form to musical attribute nodes of the musical

structures of the compound ternary form.

Fig. 4. Example of musical element. Fig. 5. Logical structure (parts formation).

42

[Analysis structure] The musical piece structure of

the results of musical piece analyses may differ for the same

musical piece due to differences in the interpretations and

in schemes of analysis of conductors. In the musical piece

model, the differences in analyses by conductors are ex-

pressed by the analysis structure.

The analysis structure expresses which musical struc-

tures attach meaning to individual granularity levels. It also

indicates the granularity levels of note sequences analyzed

by the conductor, how musical structures are decomposed

into elementary logical structures in multistage analyses by

links expressing the relationships between nodes repre-

senting granularity, logical structure, and elementary logi-

cal structure, and the relationships between the musical

structures in multistage analyses. Figure 8 shows the ana-

lytical structure of a musical piece. This analysis is done at

two levels of granularity, phrases and notes; phrases are

analyzed from the three points of view of their role, form,

and part formation; and notes are analyzed in terms of the

two musical structures of chords and meter. Some musical

structures are decomposed into elementary musical struc-

tures by multistage analyses.

[Satisfying requirements] In the musical piece struc-

ture model, the four kinds of structures determining a

performance plan of a note sequence are expressed as

follows.

x Linking a musical meaning to a note sequence in

terms of a musical structure is expressed by draw-

ing attribute-linking links to the musical piece

elements from the musical attributes of the logic

structure.

x Giving a musical meaning to a musical structure

in terms of elementary musical structures is ex-

pressed by drawing attribute-linking links to the

musical attributes within the logical structure

from the musical attributes within the elementary

logic structure.

x The inclusion relationships of a note sequence are

expressed by the �part of � link.

x Linkage relationships of musical structures are

expressed by the logical structure and the attrib-

ute-linking links.

Fig. 6. Attribute link.

Fig. 7. Multilevel attribute linking. Fig. 8. Analysis structure.

43

3. Visualization

3.1. Requirements for visualization

It is desirable that the visualization mechanism of the

musical piece structure satisfy two requirements in order to

support the creation of a performance plan, which is the

purpose of HyperScore.

The first requirement is that visualization be done

corresponding to the interests of different people. The parts

of the musical piece structure of interest differ depending

on the roles of persons involved in their performance such

as the conductor or the performer.

For example, in order to decide when to send which

signal to which part, a conductor works with the constitu-

tion of the musical piece in which the main theme is relayed

between the voice parts. On the other hand, the performer

leads the performance during the main melody, but must be

in harmony with the main melody by adjusting both tempo

and volume at other times. Thus, a performer must note the

role of his voice part and know which voice part has the

main melody. Thus, as the roles of the people involved

differ, the parts on which they focus in the musical piece

structure differ.

In addition, even the same performer must pay atten-

tion to the attributes on the level of musical elements of

large granularity in order to grasp the outline of the piece

structure, but must also pay attention to the attributes of

musical piece elements of fine granularity in order to un-

derstand the details. Thus, the parts of the musical piece

structure receiving the main focus differ with the persons

involved.

The authors consider these differences in main focus

not only as differences in the parts of the score focused on,

but also as differences in the granularity of the main focus,

the logic structures that confer the attributes, and so on.

Thus, the visualization mechanism must be able to desig-

nate the logic structure and the granularity of the main focus

in accordance with different viewpoints in order to meet the

needs of different persons and cases.

The second requirement is that the visualization

mechanism visualizes the relationships with the environ-

ment in addition to the attributes of the musical elements

themselves. In determining a performance plan for the

musical elements, there is a need for a kind of context

information which gives not only the attributes of the

musical elements themselves, but also their relationships

with the surrounding musical elements, and the nature of

the musical elements surrounding the musical elements that

are the focus.

This requirement corresponds to the visualization of

the third and fourth structures determining the performance

plan of a note sequence discussed in Section 2.1. In the case

of the inclusion relationships of the third musical element,

the performance plan is influenced not only by the attributes

of the musical elements that are the focus but also by the

performance plans and attributes of the surrounding musi-

cal elements which are in inclusion relationships. In addi-

tion, in the case of a linkage relationship involving the

fourth musical structure, a relationship which easily com-

bines tempo, volume, and so on with the melody phrase is

selected as the performance plan of the harmony phrase;

this case is an example of the performance divisions being

influenced not only by the attributes of the musical ele-

ments themselves, but also by the attributes and perform-

ance plans of the surrounding musical elements.

Thus, context information is important in determin-

ing the performance of musical elements. The visualization

mechanism consequently needs to visualize not only the

attributes of the musical elements under focus, but also the

surrounding context information.

In addition to the above two requirements, since the

musician using the system is assumed not to have computer

expertise, HyperScore has been designed by expanding a

score system which is easily understood (and readily be-

comes familiar) by a user, without using metaphors far

removed from the metaphors of a score.

3.2. Visualization mechanism

In a database, a part of the logic data structure (con-

cept schema) modeling a target score singled out in the

specific viewpoint of a specific user is called a view. Cor-

respondingly a display visualizing a part of the musical

structure selected so as to correspond to the viewpoint of a

user is referred to as a view in this paper.

The visualization mechanism of HyperScore satisfies

the above two requirements by generating a view corre-

sponding to the viewpoint of the user with respect to the

musical structure by designating four parameters: the XY

axes, display rules, selection, and granularity.

x XY axes: Designate the musical logical structure

corresponding to the X axis and Y axis of the view

display. The arrangement of the musical elements

on the view display is determined by this. For

example, in an ordinary score, the Y axis shows

the voice part formation, and if the role is shown,

a view display assembled in stages of melody,

harmony, and rhythm is generated.

x Display rules: Designate the correspondence of a

display attribute and an attribute of a musical

element. The musical logic structure which is not

arranged on the XY axes can be visualized by this

means. For example, by linking correspondences

44

between the role of a phrase and the color of a note,

the role of a phrase can be visualized.

x Selection: Designates which musical elements are

to be displayed on a view display. For example, it

can be specified that only the musical elements

having the attribute of melody be displayed.

x Granularity: Designates the degree of detail of a

display. For example, if one wants to know the

outline of the entire piece, the phrase can be the

minimal unit of a display, while if one wants to see

the details of a part, the note can be the minimal

unit of a display.

The visualization mechanism designates the musical

element under focus by selection and granularity, visualizes

the attributes under focus in terms of the positions of the

musical elements on the X and Y display axes, and visual-

izes the musical logical structure which has not been allo-

cated to the X and Y axes by using the display attributes.

Thus, a view display combining the elements of the user

focus is generated, satisfying the first requirement.

With respect to showing the context information,

which is the second requirement, the arrangement of the

attributes of the musical elements on a view display is

shown by the designations of the X and Y axes, expressing

the positional relationships of the attributes in the musical

logical structure. In addition, the musical attributes of the

musical elements can be shown by display attributes such

as the tone of a note by means of a display rule. Specifically,

a view display generated is transmitted to the user by

visualization of the context information in terms of the

display attributes and the spatial arrangement in the view.

4. Implementation of Prototype System

A prototype of HyperScore implemented on the basis

of the musical structure model and the visualization mecha-

nism discussed above is described here.

4.1. Data model

In the data model of the prototype system, a musical

structure is expressed by two kinds of nodes and four kinds

of links.

The two kinds of nodes are the region and the domain.

x The region is a node expressing the real body of

information and the classification attribute values,

and giving the musical elements and the musical

attributes.

x The domain is a node expressing the viewpoint of

the analysis, and expresses the elementary logical

structure, the musical logical structure, and the

granularity level of a musical element in the analy-

sis structure.

In earlier structured hypertext, the nodes expressing

the real body of information and the nodes expressing the

classification attribute values were separate entities. How-

ever, a node of a musical attribute is the starting point of the

attribute linking between a musical meaning and a note

sequence in terms of musical structure, and is the destina-

tion of the attribute linking between a meaning and a

musical structure in terms of the elementary musical struc-

tures. Thus, the object of the starting point and the destina-

tion of the attribute linking in the data model of the

prototype system are expressed by the same class.

The four kinds of links are: �part of,� attribute link-

ing, �member of,� and construction.

x The �part of � link expresses the hierarchical rela-

tionships between the musical attributes in the

musical logical structure, and the inclusion rela-

tionships between musical element nodes in a

musical structure and the musical attributes.

x The attribute linking is a link representing a rela-

tionship between the real body of information and

the attribute values, and expressing the meaning-

ful linking relationships.

x The construction link is a link between domains

expressing the analysis structure.

x The �member of� link expresses the relationships

of domains and the regions belonging to the do-

mains. For example, such a link connects the

domain of voice part formation and the region of

each voice part.

The data model and the classes of this prototype

system can be expressed as in Fig. 9.

4.2. System outline

The prototype system was implemented in Objective-

C on NextStep. In addition, the score system Notation [2]

on NextStep was used to delineate the score.

Fig. 9. Scheme of musical structure model.

45

The prototype system consisted of three modules: a

GUI system, a view management system, and a data man-

agement system as shown in Fig. 10.

The GUI system receives the parameters for generat-

ing a view display from the user and produces the view

display.

Figure 11 is an example of the screen of the GUI

system. Window 1 is the window displaying the musical

logic structure. The musical logic structure is shown in tree

form for each window, and the selection and granularity can

be specified. The node showing the attribute under focus is

selected and the musical element having this attribute is

presented on the view display. In addition, the granularity

is controlled by selecting either a node close to the root or

a leaf of the musical logic structure. For example, if the

child nodes of main melody, secondary melody, and counter

melody are selected without the node �melody� being

selected in the musical logic structure of the �role,� the role

of the melody can be displayed in detail.

Window 2 is the window designating the XY axes

and the display rules. The attributes selected in window 1

Fig. 10. System configuration.

Fig. 11. An example of a view (1).

Table 1. Display attributes

46

are displayed for each musical logical structure. If a display

attribute is designated by selecting an attribute among

these, the display rule relating the entity attribute and the

display attribute is established. In the prototype, the display

attributes in Table 1 can be designated.

Windows 3 and 4 are the view displays generated,

with window 3 shown at phrase granularity and window 4

shown at note granularity.

Figure 12 is an expansion of the lower right view

display window of Fig. 11 and shows different sizes of the

frame lines and the notes depending on the roles of the

phrases.

The view management system assembles a query for

the extraction of musical elements corresponding to the

input parameters by selection and granularity and assigns a

search to the data management system. It then delivers the

presentation commands assembled from the search results,

together with the XY axes and the display rules, to the GUI

system.

The data management system is a module performing

management of the musical structure and the score infor-

mation based on the proposed model. This is a simple

database satisfying search requests from the view manage-

ment system and the input and output into and out of the

musical structure file.

5. Past Studies

5.1. Performance creation support

INTERPRET [3] and Muse [4] are past efforts to

support performance creation. INTERPRET is a tutor sys-

tem which conversationally instructs beginners in the meth-

ods of performing a musical piece. The performing methods

or performance plans produced by INTERPRET are based

on musical structures. However, these musical structures

are the melody grouping structure* [1], the voice parts of

partial chord structures determining the grouping structure

are single voices, and only simple musical structures com-

pared with the complicated musical structures of the mul-

tiple voice musical pieces are dealt with. In addition, in the

function giving the musical structures and performance

Fig. 12. An example of a view (2).

*The grouping structure corresponds to the structure combining the inclu-

sion relationships of the musical piece elements and the musical forms of

this paper.

47

plans to a user, graphical displays of musical staves or piano

rolls are used as supplements to the external information on

the score, and no visualization of the musical structures is

performed.

Muse is an electronic score display for conductors

and performers of an orchestra, which presents a score

incorporated into a part of the display board. Since Muse

converts a score into part scores, it is considered to have an

internal structure consisting of voice parts, but the musical

scores for supporting performance creation are not treated

thoroughly.

As regards studies associated with the expression of

the musical structures other than support for performance

creation, 17 systems for treating individual logic structures

such as musical structures or chord structures are intro-

duced in Ref. 5. However, none of these systems treat

multiple logical structures. There are musical languages

such as SMDL [6] and MODE [7] that have functions

treating multiple logical structures. SMDL is a markup

language for music based on HyTime [8]. Although SMDL

is designed to treat multiple logical structures, no system

actually treating multiple logical structures has been re-

ported. MODE is a Smalltalk class library and tool group

for music and can create tools dealing with multiple logical

structures by using this class library. However, currently,

only tools dealing with one logical structure have been

reported.

5.2. Hypertext model

The basis of a hypertext model is a node link model

which directly connects two nodes with a one-to-one link.

The musical structure model of this paper, the frame-related

axial model [9, 10], the set-to-set model [11], and others

use hypertext models of expanded structure so as to assign

attributes to nodes and express the relationships between

the nodes in terms of the relationships between the attribute

values. These expanded models are characterized by the

ability to express multiple links in a one-to-one link model

in terms of the relationships between single attribute values

by abstracting the links by the attribute values in one step.

For example, in the frame-related axial model, the relation-

ships between the nodes are expressed in terms of order

relationships of attribute values called relationship axes

(Fig. 13). In addition, in the set-to-set model, the relation-

ships between the attribute values are expressed in a tree

structure called a facet (Fig. 14).

Among the three kinds of attribute-linking relation-

ships expressed by the musical structure model, the first

attribute-linking relationship for musical elements in terms

of musical logic structures expresses the musical elements

as nodes, the attributes as attributes, and the musical logical

structures as facets by the set-to-set model.

However, the second attribute linking for the musical

logic structures in terms of the element logical structures

cannot be easily performed by the earlier models. In the

earlier models, the side assigning attributes and the side

assigned attributes are expressed by separate kinds of

nodes. However, although the nodes expressing the attrib-

utes among the musical logic structures assign attributes to

the musical elements, conversely, the attributes are assigned

from the musical logic structures. Thus, in this paper,

regions are introduced as nodes which can assign and can

be assigned attributes, and the second attribute-linking re-

lationships can be expressed by expressing attributes in

terms of regions.

In addition, the third inclusion relationships between

the musical elements are expressed by the �part of � link

between the musical elements in the musical structure

model, and the �is a� and �part of � links can express

inclusion relationships between nodes in the set-to-set

model. However, in the set-to-set model, both documents

and subdocuments are attribute-linked from the same facet

and thus are not differentiated in terms of attribute linking.

On the other hand, in the musical structures, the musical

Fig. 13. Frame�axis model.

Fig. 14. Set-to-set model.

48

elements are differentiated from the musical logic struc-

tures which are attribute-linked by granularity. In order to

express this distinction, domains exist for the granularity

and the musical logic structures or element logic structures,

and the relationships of the regions are expressed by �mem-

ber of � links and a musical element of a particular granu-

larity is attribute-linked in terms of the musical logical

structure that is expressed by the construction link.

In musical analyses, there are quite a number of

attributes having the same names but different musical

logical structures, and expressing the relationships of the

regions to the musical logic structures in terms of the

domains also prevents these attributes with the same names

from being transferred differently from a conductor to a

performer. For example, for the tenor voice part, a tenor is

in the high register of a male chorus but in the middle

register of a mixed four-part chorus, and these factors

influence the selection of the performance plan differently.

However, different transfers of the same names occur fre-

quently. In the musical structure model, such confusion

does not easily occur, since the domains clearly express the

subordination relationships of the regions and the musical

logic structures.

5.3. Visualization

Here, the existing models and the model of this paper

are compared from the viewpoints of context information

and the designation of the user viewpoint regarding the

visualization of a structured hypertext.

Although the set-to-set model does not have a visu-

alizing function, it can extract only those notes having

specific attribute values from a node set and express the user

viewpoint by these extraction conditions. In addition, it

allows traverse operations, by which the set of extracted

nodes is gradually changed by altering part of the extracting

conditions. For example, there are four kinds of traverse

operations recognizing viewpoints: the hierarchical zoom

in and zoom out, which change the degree of detail of a

viewpoint by changing the attribute values of the conditions

to a lower level or a higher level value in the facet, and the

zoom in and zoom out, which narrow or broaden the view-

point by adding or eliminating a facet of conditions.

In the frame�axis model, a visualization mechanism

called the hyperchart, by which the nodes are placed on

graphs or table charts using the order information of the

relationship axes, has been proposed. The user viewpoint is

designated by selecting the relationship axes. Thus, a hy-

perchart* in which the selected relationship axes are placed

in correspondence with the XY axes is generated.

The visualization mechanism of this paper produces

a view display by designating a viewpoint in terms of four

parameters: the XY axes, display rules, selection, and

granularity.

The three models compared from the point of view

of the designation of the user viewpoint show the following

three differences.

Number of axes: Only three sets of relationship axes

can be visualized by the hyperchart, even if three-dimen-

sional display charts are used. However, an arbitrary num-

ber of facets can be used for conditions in the set-to-set

model. Since the visualization mechanism described in this

paper can visualize musical logical structures which cannot

be visualized by the XY axes by allocating them to display

attributes by means of display rules, it can visualize more

musical logical structures.

Degree of detail: There is no function for varying the

degree of detail in the hyperchart, and a part cannot be

magnified to receive special attention while the whole

hypertext structure is being looked at. However, the degree

of detail of a viewpoint can be designated by the hierarchi-

cal zoom in and zoom out operations in the set-to-set model.

The visualization mechanism described in this paper can

designate the degree of detail by means of the granularity

parameter.

Complexity of display: In the hyperchart, all of the

nodes loaded on the relationship axes designated by the

attribute values end up being displayed. Thus, a screen can

easily become complicated, and the parts of interest can be

differentiated only to a small extent. However, since the

attribute values of the nodes to be extracted are designated

by the set-to-set model, the nodes which do not have the

designated attribute values are not displayed. Since the

visualization mechanism of this paper also designates the

attributes of the musical elements to be displayed, the

musical elements which do not have the designated attrib-

utes are not displayed, making it difficult for a screen to

become complicated.

We next discuss the context information facility,

which gives information on the other nodes that surround

the node of interest.

Although the importance of the context information

is indicated in the set-to-set model, special visualization is

not performed for the set of nodes extracted by conditions.

Since the relationships between the nodes and the

attribute values of the nodes are expressed on a chart in the

hyperchart representation, the context information is ex-

pressed by distance relationships or positional relationships

between the node of interest and the surrounding nodes,

making visual understanding possible.

The visualization mechanism of this paper is the same

as that of the hyperchart, and it expresses the context

information designated by the XY axes in terms of distance

relationships or the positional relationships between the

*A hyperchart performing visualization by statistically processing attrib-

ute values as application charts has also been proposed.

49

node of interest and the surrounding nodes, making visual

understanding possible. In addition, the attributes of the

nodes can be expressed not only by the positions but also

by display attributes by appropriate specification of the

display rules.

6. Evaluations

In order to evaluate the prototype system, music

specialists including conductors and instructors were re-

quested to trial-use it. We obtained the following results.

x To users who have knowledge of musical struc-

tures, having studied musicology, the prototype

provides a good supporting function for the work

of establishing performance plans through an un-

derstanding of the musical structures.

x Even for people without knowledge of musicol-

ogy, the prototype is an effective supporting tool

for learning the methods of establishing perform-

ance plans or musical analyses.

x The prototype would be more effective if, for

example, a confirmation function capable of pro-

ducing the actual sound of a phrase were provided

in addition to visualizing.

In addition, not only conductors, but also composers

and performers had favorable responses to the prototype in

terms of the performance of demonstrations, and they gave

especially high evaluations to the function of emphasis-

displaying musical structures on a score and the function for

specifying the display rules of the visualization mechanism.

Thus, the following items should be improved and

evaluated in the future.

x In addition to the visualization mechanism, the

function for showing musical structures should be

strengthened by adding an audio mechanism by

allocating attributes to tone and volume.

x Since the visualization mechanism of the present

prototype system deals with only numbers 1, 3,

and 4 among the four kinds of structures determin-

ing the performance plan of a sequence of notes

discussed in Section 2.1, the linking of a meaning

to a musical structure from the elementary musical

structures by the multistep analyses of type num-

ber 2 should be dealt with.

In addition, only a tree structure is handled as the

topology of the musical elements and the musical logic

structures in the musical structure model. However, when

the pitch of the final note of a note sequence and the last

note of the next sequence of notes is the same, there is a

technique by which both notes are combined to bring out

the feeling of continuity of the flow of music. The cases in

which there are overlaps cannot be expressed by the musical

structure model in its current state. However, it is known

that these cases can be handled [1] by linking the score in

which the notes are separated before applying overlaps with

the score after the overlaps have been applied. Thus, the

musical structure model is expected to be expanded so that

overlaps can be handled. Since in the visualization, the

musician user is assumed to a nonexpert in computer utili-

zation, its expansion is designed such that it is based on a

score system which is easily mastered by the user without

using one removed from the musical score metaphor. How-

ever, applications of advanced visualization techniques,

separated from the traditional musical expressions by five-

line scores and the like, are planned as in Ref. 13, and will

not be confined in scope to expanding a score system.

7. Conclusions

HyperScore, which visualizes musical structures, is

intended as a supporting tool in creating the performance

of a musical piece of multiple voice parts. First, a musical

structure model which has been expanded from a structured

hypertext model and a visualization mechanism is pro-

posed, then the implementation of a prototype system is

discussed.

The musical structure model simply and cleanly ex-

presses the musical semantic structure associated with the

creation of a performance plan. This model is constructed

from: tree structures expressing the hierarchical structures

of the musical elements such as the phrases, motifs, and

notes; the tree structures expressing the musical logic struc-

tures of the musical form, voice part formation, and so on;

and the attribute-linking links expressing the relationships

of the above structures. The visualization mechanism dis-

plays complicated musical structures in easily understood

form by generating various views of the musical semantic

structure from different perspectives. The view is generated

by the visualization mechanism with four parameters: XY

axes, display rules, selection, and granularity. The view

illustrates the musical semantics of series of notes clearly

by their position on the view display and their display

attributes.

The logical structure of a score of multiple parts

treated in this paper has a time axis, and multiple voice parts

exist simultaneously in the structure. These features are

similar to multimedia content with a time axis, where

multiple media are present simultaneously. On the other

hand, HyperScore can treat a score on the logical structure

level. In order to make a computer into a tool for handling

multimedia, it is important that the multimedia be treated

on the logical structure level, as in the outline editing of

50

word processors. Thus, a study applying the handling of

multiple voice part pieces in terms of logical structure as

described in this study to the handling of multimedia on the

logical structural level is being planned.

REFERENCES

1. Lerdahl F, Jackendoff R. A generative theory of tonal

music. MIT Press; 1983.

2. Diener G. Ttrees: A tool for the compositional envi-

ronment. In: Pope S (editor). The well-tempered ob-

ject. MIT Press; 1991. p 157�168.

3. Backer M. Design of an intelligent tutoring system

for musical structure and interpretation. In: Balaban

M, et al. (editors). Understanding music with AI. MIT

Press; 1992. p 466�489.

4. Graeffe C, Wahila D, Maguire J, Dasna O. Muse

digital music stand for symphony musicians. ACM

Interact 1996;III:27�34.

5. Hewlett W. Musical information. In: Hewlett W, Sel-

frideg-Field E (editors). Computing in musicology,

Vol. 8. CCARH, California; 1992. p 61�134.

6. ISO. Information technology�Standard music de-

scription language (SMDL). ISO DIS 10743, 1995.

7. Pope S. Introduction to MODE: The musical object

development environment. In: Pope S (editor). The

well-tempered object. MIT Press; 1991. p 157�168.

8. ISO. Information technology hypermedia/time-

based structuring language (HyTime), 1st ed. ISO

10744, 1994.

9. Masuda Y, Ishitobi Y, Ueda M. Frame�axis model for

automatic information organizing and spatial naviga-

tion. Proc ECHT�94, Edinburgh, p 146�157.

10. Ueda M, Masuda Y, Ishitobi Y. CastingNet: Hyper-

media for organizing a visually browsing informa-

tion. Computer Software 1995;12:56�70.

11. Hara Y, Kasahara Y. A set-to-set linking strategy for

hypertext systems. Proc COIS90, Massachusetts, p

131�135.

12. Rumbaugh J. Object-oriented modeling and design.

Prentice�Hall; 1991.

13. Hiraga R, Igarashi S, Matsuura Y. Integrated perform-

ance visualization system. J Inf Process Soc

1997;38:2391�2397.

AUTHORS (from left to right)

Yusuke Ariyoshi graduated from the Department of Information Engineering, Faculty of Engineering Science, Osaka

University, in 1990 and was in the doctoral program until 1995, when he joined NEC. He currently works in the Human Media

Research Laboratories of that company. As a student, he studied multimedia systems and musical information processing. His

current research interests include information filtering.

Shinji Shimojo (member) graduated from the Department of Information Engineering, Faculty of Engineering Science,

Osaka University, in 1981 and completed the doctoral program in 1986. He then joined Osaka University as a research associate

(Faculty of Engineering Science), and subsequently was appointed a lecturer, associate professor, and professor in the

Computation Center in 1989, 1991, and 1998, respectively. His research interests include performance evaluations of LAN

access systems, performance evaluations of discrete processing systems, and discrete operating systems.

Hideo Miyahara (member) graduated from the Department of Communications Engineering, Faculty of Engineering,

Osaka University, in 1967 and completed the doctoral program in 1972. He became an associate professor on the Faculty of

Engineering Science, Osaka University, in 1980. He is currently a professor in the Department of Engineering Science Research.

His research areas include high-speed networks and multimedia. He was the recipient of a 1990 Paper Award, and is an IEEE

Fellow.

51