ENGRAVE - An Expert System that Understands and Generates ... · IntroductIon ENGRAVE is an expert...

"ENGRAVE - An Expert System that Understands and Generates Musical Notation"

by

Joseph Edward Tatem

Thesis submitted to the Faculty of ~he

Virginia Polytechnic Institute and State University

in partial fulfillment of the requirements for the degree of

MASTER OF SCIENCE

in

Electrical Engineering

APPROVED:

J. R. Sochinski

July. 1987 Blacksburg. VirginIa

L-D 5/fi ~B55 \q~1

f'1~ C,l,

ENGRAVE -- AN EXPERT SYSTEM THAT UNDERSTANDS AND

GENERATES MUSICAL NOTATION

by

Joseph Edward Tatem

Committee Chairman: John W. Roach

Electrical Engineering

(ABSTRACT)

ENGRAVE is an expert system that performs the work

or a music engraver. The system takes a digital image of

a conductor's score as input and produces

engraved-quality copies or the individual parts.

ENGRAVE consists of a vision system and a graphics

system. The vision system contains a low-level section

that recognizes features and a visual expert system that

understands their meanings. Under the current computer

vision paradigm, low-level processing is general-purpose

while domain knowledge is used only in "high-level"

routines. We hypothesize that domain knowledge is

necessary at the low level as well ror a vision system to

work reliably. The visual expert uses a frame-based

approach to explain the objects round in music. In the

graphics system, we make use of the concept or "spatial

planning" to generate correct musical notation. We show

by example that spatial planning can be applied to other

domains.

Acknowledgments

I would like to take this opportunity to thank

those who have been inv01ved in the project and have

supported me while I was working on 'it.

John

He was both

right mix.

Roach was a most able and competent advisor.

demanding and understanding with just the

Somehow. John knew when I needed a push and

when needed a break and was nearly always available at

odd times and on short notice to discuss problems and

ideas. Much of the success of this project is due to

John's insistence on thoroughness and quality work.

Special thanks to Jim Sochinski for making his

knowledge of musical notation readily available and Tor

helping to provide the detail necessary to produce a good

system. This research had its beginning when Jim inquired

into the possibility of building such a system as

ENGRAVE.

My girlfriend. then fiance. then wife. and now

soon to be the mother of my child. was a constant and

much-needed support and shoulder during both hard times

iv

and good times. Thanks for sticking with me. Dale.

would like to thank Andy Tatem, Hike Stone, and

Chrfs Philipoom for putting me up and for puttfng up with

me f'or "a couple of' weeks."

My parents went above and beyond the call of

parental duty to provide financial support. sage advice

and prayer. I deeply appreciate their sacrifice to help

me finish. Also, many thanks to the members of the home

fellowship that meets in their house for your unflagging

support.

Jesus

Most importantly. I

Christ who gave me

course."

v

would like to thank my Lord

the strength to "finish my

Table of Contents

Chapter 1 Introductfon •••••••••••••••••••••••••••••••••• !

Chapter 2 2. 1 2.2 2.3

Literature revlew ••••••••••••••••••••••••••••• 9 Printing •••••...•..•• Recogn i t Ion .....•..•. General information ••.

Chapter 3 Low Level Vision ••••••••••••••••• 3.1 Implementation .••......•.•. 3.2 Line Angle and Thickness. 3.3 Labelling the Background. 3.4 Horizontal Lines •.•..••... 3.5 Horizontal Line Deletion. 3.6 Figure Segmentation .• 3 • 7 Summary •..•.•••.••..•..

.9 1 1 12

· .20 . .. 24 . .. 24

• .30 · .. 35

· .37 • •• 41

· .45

Chapter 4 V, sua 1 Expert System ••• Implementation.

...••.•••.•.. •• 54 4. 1 4.2 4.3

An Example. Conclusion.

. •.•. 57 · .59

• •• 66

Chapter 5 Spatial Planning Tor Musical Output •••••• .•.• 72 5. 1 Planning Ahead •••••.• . . . . . . 73 5.2 Constraint Gu Idance .. .74 5.3 Description •• ..•• 77 5.4 An Example. • •• 80 5.5 Beat-Units .• · .94 5.6 Rendering ..••. · .97 5.7 Constraint Guidance In Another Doma in •• ..104 5.8 Literature Rev i ew •. 116 5.9 Conclusions •••••••••• . . . . . . .• 1 18

Chapter 6 Discussion •••••••••••••••• 6. 1 V lsi on •••••

• • • • •• • • • • • • • • • • • • 122

6.2 Graphics ••• 6.3 Conclusion.

• •• 122 ..124

• •••• 125

REFERENCES •••••••••••••••••••••••••••••••••••••••••••• 126

V I sua I ExPert Rules about Note

Appendix A A. 1 A.2 A.3

Rules ••••••••••••••••••••••• 131

A.4

Rules Rules A. 3. 1 A.3.2 Rules

Durations. about Measures .•..••••.• about Notes ••••...•.••.•• TO-FILL Rules for Notes. IF-NEEDED Rules for Notes .• about Staves ••..•••...••..

vi

131 132

• 134 • • 134

• •••• • 135 · 138

AppendIx B Spatial Planner Rules ••••••••••••••••••••• 139 B.I Constraint Guidance Rules •••••••••••••••••••• 139 B.2 Renderfng Rules •••••••••••••••••••••••••••••• 140

AppendIx C Dfagrams o~ "usfcal Characters •••••••••••• 154

AppendIx 0 NOTES ••••••••••••••••••••••••••••••••••••• 171

AppendIx E Spatial Planner Graphfcs •••••••••••••••••• 175 E.I Character Symbols •••••••••••••••••••••••••••• 175 E • 2 Beam i ng. • • • • • • • • • • • .. • • • • • .. • • • • • • • • • .. • • • • • • • • • 1 75 E.3 Ties and 51urs ••••••••••••••••••••••••••••••• 176

v, ,

List of Figures

1) Block Diagram oT the ENGRAVE System •••••••••••••••••• 2 2) Copy oT Handwritten Input to the ENGRAVE System •••••• 3 3) An Example oT ENGRAVE's output ••••••••••••••••••••••• 5 4) Output Trom [Hiller and Baker, 1957] •••••••••••••••• 13 5) Output Trom [Boker-Hefl, 1972] •••••••••••••••••••••• 14 6) Output Trom SMUT [Byrd, 1977] ••••••••••••••••••••••• 15 7) Bach's "Art oT the Fugue" as displayed by SMUT •••••• 16 8) Tenor part Trom Figure 7 as displayed by ENGRAVE •••• 17 9) Output Trom [Smith, 1973J •••.••••••••••••••••••••••• 18 10) Edited output Trom Mockingbird ••••••••••••••••••••• 19 11) Example thresholded image... • ••••••••••••••••••• 25 12) Example thresholded image... • ••••••••••••••••••• 26 13) Example thresholded image... • ••••••••••••••••••• 27 14) Example thresholded image... • ••••••••••••••••••• 29 15) Computation oT line angle •••••••••••••••••••••••••• 31 16) Line angle fmages correspondfng to Figure 11 ••••••• 32 17) Line angle images corresponding to Figure 12 ••••••• 33 18) Line angle images corresponding to Figure 13 ••••••• 34 19) Horfzontal lines ••• in Figure 11 •••••••••••••••••• 38 20) Horizontal lines .•• in Figure 12 •••••••••••••••••. 39 21) Horizontal lines •.• in Figure 13 •••••••••••••••••• 40 22) Musical Tigures remaining ••• Figure 11 •••••••••••• 42 23) Musical Tigures remaining ••• Figure 12 •••••••••••• 43 24) Musical Tigures remaining ••• Figure 13 •••••••••••. 44 25) The Tinal, thinned image ••• Figure 11 ••••••••••••• 46 26) The Tinal, thinned image .•• Figure 12 ••••••••••••• 47 27) The Tinal, thfnned image ••• Figure 13 ••••••••••••• 48 28a) A sloppily written input •••••••••••••••••••••••••• 50 28b) Horizontal lines detected in the fmage •••••••••••• 51 2Bc) Horizontal staFF lfnes removed Trom the image ••••• 52 28d) The Tinal thinned result ••••••••••••••••••.••••••• 53 29} Example oT a note Trame .••••••••••••••••••••••••••• 60 30) Diagram oT input data Tor the vfsual expert system.61 31} A more diTficult measure to understand ••••••••••••• 68 32) A plan and the notation that It represents ••••••••• 75 33} Skeleton plan Tor Figure 8 ••••••••••••••••••••••.•• 82 34) Measure 10 has been moved to the second lfne ••••••. 83 35) Notes with longer duration take up more space ••.••• 8S 36) AlloT the measures ••• now Tit on the line .••••••• 87 37) The Tinal conTiguration... • ••••••••••••••••••••. 88 38) A beat-unft ••••••••••••••••••••••••••••••••••••••.• 9S 39) The same beat-unit ••• in the skeleton plan •••••••• 96 40) Beat-units and sub-beat-units created •••••••••••••• 98 41) Beat-units combfned •••••••••••••••••••••••••••••••. 99 42) Combining level-1 sub-beat-unfts •••••••••••••••••• l00

v, t t

43) Final conFiguration .•••••.•••••••••••••••••••••••• l01 44) Conventions For spacing ••••••••••••••••••••••••••• l03 45) An example of ENGRAVE's output ••••.••••••••••••••. l05 46) An example showing ties and slurs ••••••••••••••••• l06 47) An example showing beat-units and beaming .•••••••• I07 48) A plan aircraFt carrier controller's screen ••• I09 49) The blocks Found in the controller's screen ••••••• lll 50) Initial placement of obJects •••••••••••••••••••••• 113 51) Resolution of objects oFF the screen •••••••••....• 114 52) Final screen...... ••••••• • ••••••••• 115 C-1) Whole note....... ••••••• • •••••••••••.• 155 C-2) Closed notehead.. • ••••••••••••••••• 156 C-3) Open notehead.. . •.•••••.••.•••••• 157 C-4) Sharp •••••••••• C-5) Natural .•••••••

• ••••••• 158 • •••••••••••••• 159

C-6) Flat •••• • ••••••••••••••• 160 C-7) StaFF line ••••••••.••• • ••••••• 161 C-8) Repeat ••••••.••••• • ••••••• 162 C-9) Repeat (beginning). • ••••••• 163 C-10) C-I1 ) C-12) C-13) C-14) C-15) C-16) 0-1 ) E-1 ) E-2)

F lag. • • • • • • • • • • • • • • • • • • • • • • • • • • • • ••• 164 Treble cleF....... • ••••••.•••••••..•..•••• 165 Bass cleF........ .••••••••• • •••••••..• 166 Figure "3" For time signatures. • ••.••••••• 167 Figure "4" For time signatures.... • ••••••••.. 168 Quarter rest....... • ••..•••••..•...••. 169

.. 170 •. 174

Intersection or two degenerate conics....... .178 The geometry For calculating slurs and ties .•.••. 180

E f ghth rest .............................. . NOTES Example .••..•••••.••..•••••.••••.•.••••.•

fx

LIst o¥ Tables

1) Summary of results from Figure 30 ••••••••••••••••••• 67 2) Summary of results from Figure 31 ••••••••••••••••••• 69 3) InitIally. all of the measures ••• of equal sfze •••• 89 4) Measure 5 expanded to its minimum sfze ••••••••..•••. 91 5) All measures are reduced by the same amount... • •• 92 6) All measure are reduced agafn ••••••••••••••••••••••. 93

x

Chapter 1

IntroductIon

ENGRAVE is an expert system that understands musical

notation. Its function is that of an engraver who

extracts individual parts rrom a conductor's score. It is

an expert system in that it captures the expertise or an

engraver who must be able to read a score and generate

notational1y correct parts.

ENGRAVE's input is a grey-tone digital image of a

handwritten musical score (one page at a time). Low-level

image processing routines extract primitive features from

the input image. These features are lines, circles,

blobs, and curved lines. Descriptions of them are made

availab1e to a visual expert system which interprets

these features to determine the pitch and duration of

each note as well as some articulation information. The

visua1 expert represents the separate musical parts in a

music language called NOTES. NOTES is interpreted by a

graphical expert that produces high quality graphical

output on a laser printer or other high-resolution

graphics output device. See Figure 1 for a block diagram

of the system. Figure 2 shows part of a page from a

1

2

Ol6lTA.L 1M A6E

LOW LEVEL V IS 'ON

V IS UAL EXPERT

- NOTES

SP"T'~l PLA.NNtR

,

Figure 1 Block Diagram of the

ENGRA VE Iystem.

3

~ ... .... . :-... '1.J.I) 1C "'U.-.. ~ -----T C PAR

loun4 •• v • CLARINE

ALTO SA HORN

TENOR SA

TRUMPE

T

X 5

X

T

I

TRO .. ION E I 2

I AR I TONE

lASSES

li 1'fWO

PERC .1 .. _e {em ••

'''''' I 0" "" "', )C "'&\1

,.. -......... I"'"

loll ., ,. • ...., v'

n~ .. A

I)C'Ill f(1J Utt1 ~lAJldf, l~~ V'

J ~ .. .. • I -.., II ~ n J (.(0 .. t) "--:

I ..... ·

~ • .J. .J.,. ..t .... J. .t l j

In U " • I

I ..... r _I II a-

"" ~S"11I6 f!ill-, " ~.tr:-'1 m rn r.-"\n ~~ .. .. 1& .. , .. ~: t' I ~ \..:! \.U

C, i . ~, -. T e.

Figure 2 Copy of handwritten input

to the ENGRAVE system.

.

-,,-

- - . -

.. - .. --

'" '--""

r .!~ j. .i. - - . -

• - .",

.. - .. - -.r.-;,h 'm r.1

" . t , UW

.. • •

handwritten score

shows the output

music.

4

(the input for ENGRAVE) and Figure 3

of the C-part in the same piece of

A

ENGRAVE

primary motivation behind the development of

is economics. Music copying and engraving is an

art requiring the services or highly trained experts. The

texts for copyists and engravers reveal the intricacies

of producing readable work. For example, Ross [1970]

gives almost three hundred examples of beams between two

eighth notes. That these professionals are members of

unions only compounds the situation. A publisher has a

choice between having his work done in the United States

or sending it overseas to take advantage of cheaper

foreign labor. In Korea, printing shops are run like

assembly lines where each person makes only one kind of

figure. Printing performed in this manner is relatively

inexpensive, but one must pay a price in the time that it

takes to send the inrormation between Korea and the

United States.

In the vfsion phase, ENGRAVE uses knowledge about

musical notation to interpret the input images.

Traditionally, vision systems have used domain knowledge

5

& .... ::s o

6

at high levels while low-level vision processing has

remained general purpose. During the initial development

of ENGRAVE, the plan was to use general-purpose image

processing techniques to detect musical primitives. A

high-level

was to

notation.

expert system that operated on these Features

contain all of the knowledge about musical

Many diFFerent approaches were 'tried at the low

level with little success. Finally, the decision was made

to use knowledge about musical notation during low-level

processing. The results obtained From this approach were

signiFicantly better than any other approach that was

tried. As a result or this experience, we hypothesize

that low-level knowledge is necessary For a vision system

to perrorm successrully. ENGRAVE supports this claim.

ENGRAVE uses

graphical output.

spatial planning to produce its

Spatial planning is a knowledge-based

technique that uses constraints and production rules.

Unlike its predecessors, ENGRAVE is not simply a clever

algorithm that prints music. Algorithmic approaches are

limited in capability, and systems that use them cannot

be easily modiFied or extended. ENGRAVE's capabilities

match or exceed those or its predecessors. More

importantly, any notation that ENGRAVE cannot currently

1

produce can be added with ease. This is true ror two

reasons. First, spatial planning

ror musical output that captures

problem. Any modirications to the

is a natural approach

the structure or the

system will involve

only a rew rules and will not result in changes to the

overall structure or the system. Secondly, ENGRAVE can be

easily modiried since rule-based systems are highly

modular.

Not only is spatial planning a natural approach

Tor printing music, it can be applied to other domains

that involve placement and interactions or objects in

space. Examples oT this class or problems include

production plant layout and rormatting graphics screens

ror human-computer interaction. By applying spatial

planning to a large problem, we have shown it to be a

reasible approach Tor real-world problems.

The ENGRAVE system may be logically divided into

three parts: low-level vision, a visual expert, and a

graphics expert. Each part will be treated separately in

its own chapter. In Chapter 3, we discuss the low-level

processing that supports the visual expert and the use or

domain knowledge at this level. The low-level system

8

detects primitives that are passed to the visual expert

discussed in Chapter 4. This portion or the system makes

use or rrames and production rules as a representation of

knowledge about musical planning and its application to'

musical output. A central element or the system is a

musical representation language called NOTES. Although a

rull description or NOTES is not within the scope or this

thesis, it is described brierly in Appendix D.

Most or the system is written in PROLOG [Roach and

Fowler, 1983] wIth the exception or the low-level

processing which is written in RATFOR [Kernighan and

Plauger, 1976; Krusemark and Elliott, 1981) and runs on

the GIPSY image processing system (Haralick et al, 1981].

An interrace between PROLOG and Extended CORE graphfcs

(SIGGRAPH.1979; Ehrich, 1983] was developed ror producing

graphical output.

This chapter

Chapter 2

Literature revIew

is an overview or previous work

reported in the literature on printing and recognizing

music. Each subsystem or ENGRAVE is discussed in a

separate

background

chapter.

chapter, so discussion of theoretica1

is reserved For the beginning or each

2.1 PrIntIng

Music printing programs have existed at least as

early as 1965 when Hiller and Baker [1965] reported on a

system that used a music typewriter For input. Figure 4

shows a later system by Boker-Heil (1972] is shown in

Figure 5. This system as well as many or the others in

existence uses

Genera11y, these

enter. This can

an alphanumeric language as input.

languages are diFFicult to read and to

1ead to input errors. A natural way For

humans to enter data is needed.

Besides input 1anguage, both or the systems

mentioned have two major downralls spacing and

graphics. The graphics in both systems are crude to the

9

point

is not

10

that some Figures are not recognizable. Although it

the most diFFicult problem posed by this domain,

it is an important consideration since we would like

attractive output. Proper spacing aids a perFormer by

providing a visual clue to note duration. IF a piece is

properly spaced, a performer will not have to expend

extra effort deciphering the notation [Roemer, 1973, p.

v]. Gomberg [1977] proposed a data structure and an

algorithm to provide proper spacing but neither showed

any output nor indicated that he had implemented his

techniques.

The SMUT system [Byrd, 1974] produces the best

results of the early systems. Figure 6 (from [Byrd,

1977]) shows some output. Output from a more recent

version of SMUT is shown in Figure 7 (taken From

[Hofstadter, 1979]). Figure 8 s~ows the output of ENGRAVE

on the same part.

Other printing work has been reported in connection

with musical editors. These systems generally do not

include all aspects of musical notation but leave some

tasks to the user. An early musical editor that produced

some fine output is described in [Smith, 1973] and an

1 1

example can be seen in Figure 9. The MOCKINGBIRD system

[Maxwell and Ornstein, 1984] provides the best graphics

the author has seen but leaves many notational details

For the user. Figure 10 (From [Roads, 1981]) shows some

output aFter editing to speciFy spacing, voicing and

beaming.

Some graphics work has been done For music

transcription systems [Moorer, 1977; Piszczalski and

Galler, 1977]. The emphasis in these systems is on

interpreting a waveform rather than on graphical output.

2.2 Recognition

All previous work relating to the automatic recognition

of musical notation has dealt with printed music on1y.

The best known of these systems is DO-RE-MI [Prerau,

1971;

that

their

where

music,

1975]. DO-RE-MI is a pattern recognition system

classifies the objects in an image according to

size. This technique works well for printed music

figures are uniformly printed. In handwritten

this technique is not reliable since the the

appearance of a given type of note may vary widely within

a single piece.

12

2.3 General fnrormetfon

For inrormation about the rules or conventional

musical notation, (Roemer, 1973] provides motivation for

correct notation and is oriented towards copyists.

[Stone, 19801 is intended to be a standard for unusual

notations that occur in modern music but also covers some

of the basic conventional notation. A book for engravers.

[Ross, 1970], is a clear and well-organized source

presented in a form that is easy to convert to a computer

program.

13

.J., ...... • " r ,,~ ~ J... ~ .. -.. lIP , •

• ... fI' .. -

IP

- ... J...

• ... .. ''/ • IP

._--" ,.. -

.L I · - "::=::; - r:. .. .. .. ./ ~ T ", p .. ..... "-. ,

_-j ::::==t=='=- .~ '"_·c-.. ./ P ;

'- tr. II' ,... I. · "-:::=

./ v p ,

",

I.- I'l .... L..h . · " "D ... D ..

Ip : ./

J. - ." .. .. ...J.. ... ....... c=:.:~; .. ....... , , ,,--~ .. •• • • . .. • ... ' ... . • • .. • • ...

" L L -t::--

-- "" ....

" "-., , , fj j ..., III " fI ~ fj , ijW " ii "

Figure 4 Output from [Hiller and Baker, 1957].

15

STRING aUFlR1ET

[!'lEse. DIM.

p (RES( •

1 7 I' II p

:j r it r Ir I

Figure 6 Output from SMUT [Byrd. 1977].

~~

"" tl

01

...

-~-

fI

tl

,;

_-0

.;

:l

II

16

Ii!!!!! -

~

_I-. - ...---I--~----...

...J -I --

- • • .-A-;" .---- "'--

- -- Ioo",J I .. .fIL .... - --

- I

-- .~~ Ii _____ •••

-- - --

~ I

... • , ... tt .. r- •• tt_ ...

( 1685- J750) I

Figure 7 Bach's "The Art of the Fugue"

as displayed by SMUT.

:~

- ~:- : ~--=

- - --".I)!""..~ '-_r .. .~~~~-.. --.......:

--

... -=~

----+ __ .I

;';-III~.? "ff.ii ~>-- .. ii III

I

--..

.!!..~- ... ... .. -

(

(

(

(

17

OBOE

ClU' J N£l IN 8_

BASSOON

~

I'" JI..

,'" 1_

.....

'. --"

I" ..........

,..,..,

"-

18

TRIO FOR WOODWINDS I. FOR "ADELEIN£: ANO DARIUS IHLHAUD

.... DOI.tl ;-1 ~

It 1 -==L i"

~ OOlCE' r -

........ 1

~P I

==t p J .. ~-- 't!/

-- ,~ ... ~ .. - .... .. • TfIlflO

- - 1- .. l IOOtO STtll.~t '11.1 ttfSC, _ .... ~ -'_"'-::- '-~ ......

"IU tlltSC - , F .,\ POtO ST.INl:t - . - '- ..... -

Figure 9 Output from [Smith, 1973].

~

LELAND SMITH

SUlITO ....

I

DIll, I

DIll, I

.IOOtO tUSt.

-1

J .- J ,

,.. j .J ti _

)

rn "11

~ -------:. ~-

3 3

m m

19

.. t

r-- -I""""'"

I

Figure 10

•

m

Edited output from Mockingbird.

Chapter 3

Low Level Vision

Essential to any vision system is some kind or early

processing that can provide inrormation in a usable rorm

to higher level routines. In ENGRAVE, low-level routines

detect primitives that are passed on to the visual

expert. Our primitives are the commonly occurring shapes

in music: circular blobs (ror closed noteheads), circles

(open noteheads), horizontal lines, line segments (not

horizontal), and arcs. The location and orientation data

ror each primitive are made available to the visual

expert system through a rile. Currently, computer vision

1s thought or as being divided into two areas of study:

low-level and high-level computation. Under the current

paradigm, low-level vision consists or general-purpose

operators that extract information rrom images.

High-level processing interprets this inrormation using

knowledge about fts domain [Ballard and Brown, 1982, pp.

2-6].

During the fnitial phase or development, ENGRAVE was

designed under the current paradigm from computer vision.

Many or the standard techniques ror line detection were

20

21

tried with little success. These techniques could only be

made to work consistently for a small class of images.

This is particularly true of handwriting where there is

large variation between one person's handwriting and

another's. When the system parameters are adjusted for

one set of images, we find that the operators do not work

well for another set. It is difficult and often

impossible to adjust the parameters so that a system will

work reliably for all classes of images in a single

domain. Eventually, we concluded that general-purpose

techniques would not work and that explicit use must be

made of domain knowledge. Concerning the segmentation of

curves, Rosenfeld and Johnston speculate that "if the

user has special knowledge about his class of curves, he

might well be advised to devise his own segmentation

procedure taking this knowledge into account" [Rosenfeld

and Johnston, 1973]. An expert system for low-level image

segmentation is described in [Nazif and Levine, 1984].

The knowledge in their system is about low-level image

processing. We propose not only the use of image

processing knowledge, but of domain knowledge as well.

When domain knowledge was used at the low-level in

ENGRAVE, the results obtained were remarkably better than

any results obtained under the current computer vision

paradigm.

The

provided

22

horizontal staff lines in a musical score

a particularly difficult problem which was

solved by incorporating knowledge about musical notation

into the low-level process. Staff lines intersect most

figures in a piece of music. Should the intersection

pixels be labelled as part of the staff line or as part

of a figure? In reality, these pixels are members of both

the staff and the intersecting figure. Most labelling

schemes, however. can only label these pixels with one

label. Prerau [1971] simply removes the staff lines from

the image thereby causing notes and other figures to be

broken into pieces. These pieces can be easily

reassembled because of the regu1arity of printed music.

With handwritten music, we cannot make use of such

regularity. Instead, we make use of knowledge about the

domain first to detect horizontal lines and then to

remove them intelligently so as not to lose any

information.

The Hough transform has been used successfully in

many applications to detect straight lines. circles, and

ellipses [Duda and Hart, 1972; Tsuji and Matsumoto,

23

1978]. However, the Hough transform has only been used to

detect a few figures at a time. On a single page of a

conductor's score, there are more than 50 stafr lines in

addition to the lines for note stems, bar lines, etc.

In' this application, the Hough space is too noisy

for us to be able to detect all of the figures reliably.

Less prominent figures are obscured by the stronger ones.

Watson et ale [1984] have developed a line tracker that

functions well in a similar domain. The tracker uses

grey-tone values to determine whether to a assign a pixel

to a line or to start a new lfne. Because of the many

line intersections, the line tracker tends to turn

corners or to break notes and other musical figures

apart.

Closed noteheads can be detected as convex blobs. A

spoke filter [Minor and Sklansky, 1981] was tried, but it

detected the line intersections as blobs as well as the

noteheads. In some cases, the intersections were stronger

than the notehead. Haralick [1974] gives a circularity

measure that is simple and works we1l when used with a

threshold. The region detection portion or Watson's line

tracker also works well detecting these noteheads.

page

image

24

The input image is a grey-tone dfgitization oT one

oT a score. This image is converted to a binary

using the double-adaptive threshold (OAT) oT Watson

et al (1984]. Figures 11 through 13 show some samples aT

an image aTter thresholding.

ATter thresholding,

perTormed to determine

some preliminary processing is

line angle and thickness. Using

this inTormation and knowledge oT our domain, horizontal

lines are detected and removed Tram the image. Once the

staTT lines are removed, we are leTt with an image where

the notes and other Tigures are all isolated. These

Tigures are broken into their component line segments.

AlloT the inTormation Tram this low-level system is

passed to the high-level expert through several Tiles.

3.2 Line Angle and ThIckness

Line direction and thickness are important

propertfes that need to be computed. Line direction gives

essential inTormation to help determine horizontal line

25

1~~-~~~~1--~~~--~~2----~-~~-3---~-~---4~------~-~1 I J I J J J J J J J J AAAAAA J I AAM,A,.4.AAAAA J J AAAA J J AM J J AA J J A AAA A AA J IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAJ I MMA /lit itA J I AAAAA itA itA J I MAA itA M J J AAAA A AM J J AAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ 2 M AAAA 2 I /It AA J J /ItA ,. AA J J AA A AA J I AAA AAA A AAAAAA AAAAAAAAAAAAAAAAAAAAAI JAA~~~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ J AA AAA AA J J AA AA AA J I itA AM M J I AAA /lltAA AA I 3 AAAA AAAAAA /lit AAAAAAAAAAAAAAAAAAAAAA3 JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ J MM /lit A J J J I J I _ I J /lit AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ JAAAAA~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I -I J J 4 4 I J 1---~~-~-1~~-----2~--~---~-3--~---~~-4--~---~--5-J

Figure 11

Example thresholded imaae Irom the handwritten Input. An ei&btb Dote beamed to • half Dote. The bottom of tbe eiabth Dote tail coincides with the .taff line.

26

J-~~~~--~-1--~---~--2~~~---~~~3-~~-~--~-4----~--~-S~J J I J I J I J J J J J I I AAA J J AAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAI IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAl I AAAAAA AAA AA 1 J AAAAA IttA IttA I J AAAM A.AAA J J AAAAAA Itt AAA J J AAAAA AAA AAAA I JAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAI IAAAAAAA AAAAAAAAAAAAAAAAA AA IttA AA Itt AAAAA I J AA AA I J itA AA I J M AA I 2 . M AAA ~ JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA It AAA J J itA "A I I AA AA J I AA AA I I AA AA I JAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I ~"JIt. ... ".AAAMAAAAAAAAAAAAAAAAA A It A I I Itt I 3 M 3 J Itt J I I IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AI I I I I J I J J J I 4 4 I I 1~-~~~----1--------~2--~-----~3--~---~--4-----~~--'-J

Figure 12

Example tbresbolded Imale from tbe lIaDdwritteD iDput. A quarter Bote followed by • lIalf Dote. Tbe top of tbe balf Dote tOiDtides witb tbe top Italf liDe

27

1~-~~~~~~1----~--~-2~--~--~--3------~--4--~------~~J I M I I AA I I AA I J I J I I A J I AA J I AA I I AA J 1 AA A AAAAAAAAAAAAAAAAI JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JAAAAA AMA A A AAAAAAAA A I J AA AA AA I J AA AAAA AA AAAA J J AAA AA AA AAAAAA I J AA AAA AA AAAAAAA I JAAAAAAAAAA~·AAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAI JAAAAAAAAAA A A AA AAAA AAAAAAA J I A A AAAA AAAAAA I :2 . ItA AMAAA AAAAA:2 J AAAAA AM AAA J J . AAAAAAA AA AA AAA J JAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAI J MAAAAAAAAAAAAA AAAAAAAAAAAAA AAM AA I J AA AAA AA I J AA AA AA I J AA AA AA J J AAA AA J JAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI 3AAAAAAAAA.4:AAAA AAAAAAAAAAAAAAAAAAAAA AA AAA 3 I A A J J A I I J J I IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI IMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAA A J I· J J I J I 4 4 J I J-----~-1--~------2--~~---~-3--~--~~~-4--~------'-I

Figure 13

Exam pie tbresholded Imale from tbe baDdwritteD iDput. Quarter rest, Datural, quarter Dote with dot above It. Note break In starr liDe Dext to rest; also, the Datural bas ODe discoDDected tail part, a Itaff liDe throulb its middle, aDd its top borizoDtal IiDe coiDcides witb a .taff IiDe.

28

pixels and thickness can aid in the identification of

figures that cannot be otherwise identified. For

example, many closed noteheads in our fmages are so small

that they cannot be detected reliably by the circularity

measure. Our eyes, though, can discern their "presence

(See Figure 14). These figures can be properly identified

later by observing that one end of the stem is thicker

than the other.

To compute this information, an odd-sized circular

window is passed over the image. If the center pixel is

black, then the window is scanned to find the black pixel

farthest from the center. ,

The angle of this point and

distance from the center define a vector that

approximates the line direction at the center pixel. To

compute line thickness, the window is scanned along a

line perpendicular to the direction vector until a white

pixel or the edge of the window is encountered.

In many cases, a straight line between the farthest

pixel and the center will cross white space. If this

occurs, the angle will not correspond to the line

direction. To prevent this from occurring, some pixels

are deleted from the window before computing the angle.

29

1~~~~~--1--~~~~---2-----~---3--~~~~--4--~~~-~--5-J J AAM"A J J AAAA I I AA I J J I I JA AM I JAA~~~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I It It AAAAAA AAAA AAAAAAAAAAAAAA AAAAAAAAAAAAAAAI J ~A J I A~ 1 I MAA I I ~A I JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI JA AA·~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I A.4.AA.6.A AAA J J AMMA MA J I AAAA A AAAA J J It AMAAA AAA I JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAMI 2AA~~~~~~~AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA2 J Mit AA J I It It itA J J itA itA J I It AA I IltAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA A AAAAA AAI JMAM.,V~.AAAAAAAMAMAAAAAAMAAAAAAAAAAAAAAAAAAAAAI J itA I I A I I A J 3 3 IAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AA A I JAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAI I I J I J I J I I I I I I I 4 4 I I 1~~~--~--1~--~----2---~~~--~3----~~--4-------~-~~1

Figure 14

Example tbresbolded Imale from tile .andwritten loput. UslOI a malDifyiol &Iass, we were able to determine that this result was Dot uDfaithful to the orilioal. Dilitization of oriliDal pale was 1200 x 100 (horizontally, therefore 100 IcaD lines per incb). The blob to the left was meant as a sharp, the filure to the rilht is a quarter Dote. The quality of this iDput is Dot typical.

30

Any black pixel that (1) is not the central pixel, and

(2) is adjacent to a white pixel that is closer to the

center than itselr, is deleted. This process is repeated

until no more pixels are deleted. Figure 15 illustrates

this process. Arter deletion, the line angle and

thickness are completed.

To detect horizontal lines, the radius or the window

should be set to a large value -- at least halr the width

or one starr in the 'image. This will Torce pixels at the

intersection or a staff line and a figure to have a

horizontal line angle. Figures 16 through 18 show the

line angle images computed from the examples.

3.3 Labelling the Background

Many figures such as half notes and accidentals have

"holes" that iF detected can be useFul in identirying the

Figures. To this end, the image background is segmented

by a simple labelling process as follows:

- Ir an unlabelled white pixel has no labelled white a-neighbors, then give it a unique label.

- IF a labelled white pixel has labelled a-neighbor, then label pixel with its neighbor's labe1.

a differently the original

(¢~" ... J.'<V ~.Q.. \ .e,.. \:, ot/\

-

'figure 15 Coll1Vut.ation of line an~e.

/

\.

I

I

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

32

~~~lf ., ., " '- 'v""~ ~~~--,)'" ,,~~~~ .......... t""t"" ..... ltr r """" • It • \I" -:

~ • -"

, "

CI) ...

" •• ~ """ :::s 0.0

" ~ , """" i.t:

~ "-" 0 -» » i "" 0.0

i i ~ ,. .. itf' it "" r:: ;s

i i~~lI;itll;i tt '" \C C

i t 't~~i-- "" - 0 0.

i i Iti" ii "" CI) CI.l ... CI)

:::s t i i • i""" ii "" 0.00

i T i" i iT '\, i.t: C)

CI.l

t t i" i iT 'l'\ CI)

0.0

l i i'\ it IT '\ co

it i ii~rr i '\ .5 T i it ~ii ~~~ii

CI)

t ~ ~~~~~ ~~"4-Ci;,

~t~~~~~t ~~~~tt ttlt a CIJ

i it i ~ii~ ii c

i i t it»-- iiw ::i i i i+ ii~------ti~ it iJ, i+ ii ~~ i ii iJ, iJ, ii ii ii iJ, iJ. ii ii ii it iJ, ii i it it

++ ii " i ii it it " i ii it if ii 'i

i't it it ii ~ i it ii it ii .\0 i ii it ii it 'i ii it iT ii f i ii ii ii it ., t tf 1'i ft ii t i Ti ti if tT , f if if iT ii 1'i

33

\,J, J, J, J, J, • 'oaIJ, J. J, J, J, ~J, J, J, J, J, ...

~± J, ± : J, J, • J, J, .1 '" .... ..l. ... '.11~ ..l. J, II J, J, '"

...,., .... J,.,. .... '.II .... '" .1 J, .1 '" lIfJ, "'~"'J, J,rr J, J" lItJ, ~~+ )J, J, J, ,

'.1 ~~~~~~~J,±~~~~J,~ ± ! tJ,~ ~~~~~~J, ~~~~J, J,01 ~ )t~ J,J, J, J, ,

J,J,~A~~ ... t~ J,J, J, J, J,J, ~1~ .1..1. J, .1.J, J,J, t J,.1. J, J,J, M J,J, l' • J,J, t · J,J, -J,J, J,J, J,J, G) .... J,J, 4 J,J, J,J, :::s

CO

±± 4 ±t ±± ti:: 0

J,J, J,J,

it ...

I • J,J, i±

co 4 =

J,J, J, ~ r- =

1.1 J, ~ J,J, .1 J,J, _ 0

c. 1J, J, • .1J, J, J,J, G) CIl .... ~ 1J, il J,J, J,

,I t =' I:: coo

Ii J,J, J, ti:: Y

J,~ J,J, ±

CIl G) co

J,l J,J, CI:I

rJ, J,-l. J,J, ±

.5 J,-l, J, G)

I J,-l. J, -;0

s J,~ J,

G)

J,-l. J, = J, , J, :.:J

'" , '" J,1' f t1'

if '.II 1'1' J, lI~\rt f • rr -+ -+-+-+-+-+ t-+ ....... lt'" ~ .... ~~~+-+-+--+ -+-+-+-+--' l~~~"" .... 4f" ........ ~~ .. ..... ~'" , .. "A~~~

, ~ ~ ~ '" t ~ 11\ t ,. ~ t ,. It t ,. It t 1 It t

• ...

t"" :r (II

§ (JQ

(i'

9' I»

~ I:n

n :!t OQiQ .., = (; (; I:n

"0 o -:s co Q.

S' (JQ

... o ~

OQ' = .., ('II -~

j i i

+t tt ii if tt iT tt ii ii

i ~ i po i ,. i

i i i i

· i i i i i i i i

++ ++ ~I · ~+ +~ ~ " ~~ ~~ ~ . ++ ++ ~ ,. .l.J. J,J,'''- ,. ,.., "~"' .. lt4 '''' tit "'t,.,. It''' A4'1t ... 11,." • fot Aot"lIt f ..,~~ ...

" "'''I'' .... t t t t ,.,,. . . ~

ttc ttl' ~tt~~,.~ttl' •

tt""",."ttl' t tt t t

" " " 1- • + ~ t -i J, 1t i i J, ¥ • j f 1 ~ i i 1 ~ i i 1 .. ~Ttlltllt.i 1 t ~~~~t~~~~~i~~~~~ r t .... It illt 4~~f"'~~~ r t ~lIlt i • ~i r ~"~i·i r ""~i 1i T .. It". ......... i ... ~~~~i~ i ~ .... ~". .... ~i~~~~~i if ~~~,.1 i i ii "i i iJ. iii iJ,

.. 1 i f fJ, poi i f iJ,

f t i jJ, iii i' f i i i~ i i ""i~ i~ i i ~"lIti~~ i~ i i"~~~~f~~~ it

~f~~~~~i •• ~~~i~~~ ·i~ ~~~T~~~~~i~~~~~i~~~ it

~i i f~i"''''''' i~ i i fi~'" i~ iii i~ i i f i~

vE

• .. . .. ( .

35

This process is continued until no more labelling is

done.

The difference between the "holes" for which we are

looking and the ,other white areas in the image is that

the holes are significantly smaller and they are located

inside of musical figures.

3.4 Horizontal Lines

The most important figure to deal with in music is

the horizontal line. Nearly every figure in a piece of

music 'intersects a staff line. Our strategy is to detect

horizontal lines and then remove them in a manner that

does not break up the other figures. We use angle

information and pixel adjacency to determine whether a

pixel is a line pixel. The rules are as follows:

A black pixel is marked as a horizontal line pixel if'

it is to the left or right of a line pi xel; or it is horizontal and it 8-adjacent to a 1 f ne pi xe 1 •

If a pixel is horizontal and has no marked adjacent pixels then mark it with a new label.

A white pixel is a horizontal line pixel ,if it has a marked pixel to its left and to'its right. (This rule allows for the possibility that a line

36

may have a i-pixel break.)

Orten, it is impossible to tell by its neighbors

whether or not a pixel fs part or a horizontal line. A

pixel may even be both part or a horizontal line and part

or some other rigure. These pixe1s are given a special

label that indicates that these pixels are

"questionable." That is, it is uncertain whether these

pixels are part or a horizontal line or not. A list or

these pixels is stored in the output rile ror use at

higher

pixels:

levels. Here are the rules ror questionable

A pixel is questionable ir it is not horizontal and 1s above or below a line pixel.

A pixel is questionable ir questfonable pixel and questionable pixel.

it is below

above a another

- a questionable pixel above a line pixel is re-labelled as part or that line ir none or the a-connected pixels above it are black.

- a questionable pixel below a line pixel is relabelled as part or that line ir none or the a-connected pixels below it are black.

As pixels are labelled, they are added to a list or

pixels that all have been marked with the same label. A

list is also kept or questionable pixels.

37

Some pixels may be elements both or a line and the

questionable list. These will generally be pixels that

are both part or a line and some other Figure.

AFter all the line pixels are identiFied, there

usually are adjacent regions that have been labelled

diFrerently. With the exception Or the pixels that have

been labelled "questionable," these regions are part Or

the same line and are merged.

A least-squares Fit is used to determine the

equations Or the lines and these coerFicients are made

available in a file along with the list or questionable

pixels. We can identiFy starr lines by the ract that they

are almost as long as the width of a page and that the

other horizontal lines are much shorter. Figures 19

through 21 show the line pixels that were detected in the

original images.

3.5 Horizontal Line Deletton

Once the horizontal lines have been detected, they

are removed rrom the image leaving the other Figures

38

1~-~---~-1----~--~~2-~--~----3----~---4----~~---S-J 1 J I J I J 1 J 1 J I J 1 J J J I J 1 1 JIB B IS J ICCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCJ ICteeeceeeececccececcccccccccccceceeeecececcccccCCCCI 1 J 1 J I J I 8 I 1 IBBS RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRJ JRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRJ 2 8 B B 2 I J I J J I I B ISB B IB BB8 QQGGQgQQQgQGGGGGQQgGGI IQQQogQQgcQogQggQQQQQQQOOBQgQgQggQQQQQQQQQQQgQgggQgQJ I B J I J 1 I 1 I 3 B BBB I 44444444444444444444443 1444444444444444B48B444444444444444444444444444444441 J BBBB BB J J J J J J I I B '~~~5~5~"~~~~555'55'5""""I JSSS5~SS"SS'55'5"55'5"55"""'5""""""""51 I . I J I 4 4 I I 1~-~--~---1--~------2--~~--~--3~-~-~--~4----~----5-J

Figure 19 Horizontal lines, staff lines,

extracted for image in Figure 11.

39

J~-~---~~1-~-------2------~~~3--~-~---~4-----~-~~5-J J J J J J J J J J J J J J BBB J J CCCCCCCCCCC CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCI JCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCI 1 8 8 1 J J J J J J J 8 B 8 J JOQQQOQOOOOQQOOOOOOOOOOOOOOOOOOOOOOOQOBOBOOOOOOOOOOOI JBBBBBBB OOQQOQOQOQOOOOOQO IB IB BI I BIB J J J J I J I 2 I 2 10000000000000000000000000000000000000000000000000001 JoooooOOOOOooOOooOOOOooooOOOOooOOOOooOOOOOO I IBB J J J J J J I J J JVVVVYYYYYYYYYVVVVYYVYVYYVYVYYYVVVYVYYVYYVYYYVYVYYYYJ JYYYYYYYYYYVYVVVVYVYYYYYYYYYYY B B B J J J 3 3 J I J J JUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUI JUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU BI I J I J J J J J J J 4 4 J J J~~~~~~1---------2-~----~~-3---------4-----~~--5-J

Figure 20 Horizontal lines, staff lines,

extracted for the image in Figure 12.

40

1~~~-~-~~1~-------2~--~--~--3---~---~-4-~--~----5-1 I I I I I J I I I J I I I I J J I I 1 A BBBBBBBBBBBBBBBBl JBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBI ICCCCC C C C C c ceaaa C I I I I J I J I D D J IEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEI IFFFFF FFEE F F F EEEEEEE I I F I 2 2 I J J QgQ QQ I JHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHI J HHH-IHHf+IHHI+I I I I I 11 I I I J HHH J r I I J J I I I J I IKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKI 3KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK LL L L 3 J L J I J J I I I 1"~KMMMM""MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM""MMMM"""""r 1""t9V1~MMHJ1M~I1MMI1HMHHM NNN N I I I I J J I 4 4 J J 1-~-~--~--1~~~-~---2--~------3------~--4---~-----5-1

Figure 21 Horizontal lines. staff tines.

extracted for image in Figure 13.

41

intact. This is not done by deleting line pixels, but by

marking the pixels that should remain. The rules for

marking follow:

Mark a black pixel if - it is not a line pixel; - it is not horizontal but the pixel above

or below it is horizontal; - it is a line pixel and is above or below

a marked pixel; or - it has marked pixels to its left and to

its right.

When this process is completed, the musical figures will

have been isolated. Figures 22 through 24 show the

results of line deletion.

3.6 Figure Segmentation

Once the figures have been isolated, they are

segmented into their components and this information is

made available to the visual expert. First, the figures

are thinned using the algorithm given by Arcellf and

DiBaja [1981] in order to obtain chain codes. Their

thinned figures are segmented at junction points.

Junction points are defined as pixels where three or more

segments having the same label meet. After segmentation,

we are left with the chain codes of all of the arcs in

42

J---------l----~~---2--~~~--~-3-----~---4--~~---~-~-J J J J I J J I J I I J MAAAA I J AAAMAAAAAA J J AAM J J AAA J J AA J JAM A J I MAAAM A AA J I AAAAAM A AA I J MAAA A AA J J MAAA AA AA I J MAA AAM I I MAA AAM I I AAAA AAAAAAA J I MAA AAAAAAA I 2 ItA ItAAA 2 I A AA I J M A AA J J M A AA J J AM AAA AA J I AAA AAAA AA J I AA AAA AA J J AA AA AA I I M AAA AA I I AM AM AA J 3 M" MA litA 3 J MA AM AA J I M J I J I J I I I J I J I J I J 4 4 I J 1~~-~-~~1----~~--2-~-----~3--~----~-4-------~~~-1

Figure 22 Musical figures remaining once the

horizontal staff lines have been intelligently removed from the image in Figure 11.

43

J~--~-~~~-1~--~~~~-2-~-------3--~~~-~-~4-----~---5-1 J J I J J J 1 1 J J 1 J 1 'A J I AAAAA A AA J J AAAAA AA AA J I AAAAA AA AA 1

·1 AAAM AA AA J I AAAAA A AAA J I AAAAAA A AAA J I AAAAA AA,AAAA J I AAAAA AAAAAAA J J AMA AltAA 1 I AA AA J J M M J J AA AA J i2 AA AA 2 J AA itA J J AA itA J I AA AA J J itA itA J J itA itA J J AA itA J 1 AA itA J 1 AA It J J A J 3 M 3 J A J J J J J 1 J I J J J J J J J J J 4 4 J J 1-~--~~--~1--~--~~--2~~-~---~-3--~~~---4~--------S-J


horizontal staff lines have been intelligentl y removed from the image in Figure 12.

44

J- --J---------~---------~---------4---------~-l J J 1 AA J 1 AA ) 1 1 J 1 1 A J J AA 1 1 AA 1 1 AA J J AA A 1 J AA AA AA 1 J AA A~ AA J I AA AA AA J I AA MAAA AA AAAA J J AAA AA AA AAAAAA J J AA AA AA AAAAAAA J J ~AA AA AAAA A~AAAAA 1 J AAA AA AAAA AAAAAAA 1 J A A AAAA A~AAAA J ~ AA AAAAAA AAAAA 2 I AAAAA AAA AAA 1 1 AAAAAAA AA AA AAA J J AAAAAAA AA AAA AAA I J AAAAAAA AA AAA AA 1 J AA AAA AA J J AA AA AA 1 1 AA AA AA J ) AA AA J ] ' ... A AI\ J 3 AA AA 3 J A A 1 1 A J J J J J J 1 J J ] J J 1 Jl ~ 4 J J J---------1---------2---------3---------4---------~-J


horizontal staff lines have been intelligently removed from the image in Figure 13.

45

the image. Each individual arc is Further segmented by

breaking the arcs at curvature maxima as described in

RosenFeld and Johnston [1973]. Figures 25 through 27 show

the results of thinning and segmentation.

Descriptions of all the features detected are output

to a file. This rile is loaded by the expert system which

uses it to identify the figures.

3.7 Summary

The results obtained using these techniques were

quite good

cases, the

segment the

available to

-- certainly far above expectations. In most

system was able to correctly isolate and

figures, and enough information was made

identify them correctly. In portions of the

score where handwriting was sloppy, some information was

lost. Figures 28a through 28d show successive results of

processing a sloppy image.

Originally, our plan was to use general purpose

image processing techniques as early processing for a

visual expert system. It was round, however, that

general techniques could not be made to work we1l.

46

J~~~~~-1~--~-~--2-~-~-----3----~----4---------'-1 J J J I I I J J I J J J J AAAAAAAA J J itA J J It I I A 1 J A J I It D J IDE J I B D E J J B 'D E I J B D E J J B D E J J B D E J J B DDU J 2 B F ;l J B F J J B C F J I B C F J J B C F I J B C F I J B C F J J B C F J I B C F I J B C F I 3 F 3 J F J J I J J J I J J -I J J I J I I I 4 4 J J 1~-~-~~~-1~~-~----2--~---~--3-~--~----4--~----~-S~J

Figure 25 The final, thinned image corresponding to the image in Figure 11. Other information

including where blobs (noteheads) exist and their extent~ candidate horizontal lines connecting

music figures. etc. are passed to the high level expert.

47

1~-~~-~~~-1~---~---2~-~-~-~~-3---~~~--4~~---~~--S-J I I I I I J I J J J J J J J J J J A J 1 DBA 1 J DBA J J DBA J J DBA J J D 8 A I I EF BAAA J J F C J J F C J J F C J J F C J 2 F C 2 J F C I J F C I J F C J J F C J J F C J J F C I J F C J J F I I F J 3 F 3 J J J J I J J J J J J J J I J J J I 4 ~ J J J~----~---1~-----~-2-----~---3~--~~-~-4~----~---5~J

Figure 26 The final, thinned image corresponding to the image in Figure 12. Other information

including where blobs (noteheads) exist and their extent. candidate horizontal lines connecting

music figures, etc. are passed to the high level expert.

48

1--~~---1~~~~----2----~~---3--~--~---4---------'-J J I J A I J A J J I J I J I J I J B J J B I I B 1 J B J J F Bel J F B C J J F Bel J F Bel J F Bel I F Bel J FIe Q I I F B C Q I 2 F DeC Q 2 J F D g I J FF . D Q I J FFF Q I J F E Q I I F E Q I I F E Q I I F Q I J F Q I I F Q I 3 F Q 3 I Q I J I I J I I J J J I I I J I J J 4 4 J I J~~~~~~1~~---~-2--~-~-~--3--~~-~~--4-~---~~--'-I

Figure 27 The final. thinned image corresponding to the image in Figure 13. Other information

including where blobs (noteheads) exist and their extent,. candidate horizontal lines connecting

music figures. etc. are passed to the higb level expert.

49

Domain knowledge had to be incorporated at the low level

to achieve satisfactory results. This is an important

result since vision systems typically do not explicitly

incorporate knowledge

Traditionally, world

during

knowledge

low-level

is used

processing.

only ror

"high-level vision" and low-level processing is done

using general purpose mathematical models.

Much or the lower-level knowledge in this system is

procedura1 not declarative. Where it was used. the

rule-based approach gave the system the flexibility it

needed to do its Job. Several of the processes in this

low-level system were implemented under the old

general-purpose paradigm. The system would benefit rrom

also making these processes rule-based and rlexible. The

importance of this approach is not that we have achieved

the ultimate system but that a dramatic improvement

resulted from adding world knowledge to low-level

processing. This suggests that a vision system must

incorporate world knowledge at all levels to work

successfully.

50

1---------1---------2---------3---------~---------5---------6---------7---------1

I A AAAA I I AAAAAAAAAAAAAAAAAA I I AAAMAM. I M~A J I AAAAA • I AA AAAA I I AA AAA J I fIlA MAA J I A' A N!AA " I I ,,~~ :~ :! I I "AM AA A AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJ IAAAAAAAAAAMAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAMAAAAAAAAAAAAAMAAA AA I I A AAAM M AAAAA AA I I AAAA AA A AAAAA MA I I AAAA M"" A AAAAAA A AAAM I • AAA A AA MAA AA M AMAA I a . AAA AAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAMA~ IAAAAMAAAAMAAAAAAAA""MAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAA""AAAAAAA AAAAAMAAAA I • A AA AAAAA AAAAA AA AAA AAAA I I AA MAAA AAAAA MAAA AAA 1 I" AAAAAA AAAA AAAM AA I I A "AAAA A AA AA AA I J A AAAAA AA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAJ IAAAAAMAAAAAAAAMAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAMAAAAAAAAAAMAAAAAMAAAAAAAAAAJ 1M AA All. AA AM J 1 A AA A AAAA I 3 AA AA A AAAMA 3 I AA AA AA' A AAAAAAAA I IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAI I AAAAA M MAA M A /ltAAA AAA AA " I I ItA AA AA • I M AAA AA I I MA AAAA A A I J MA AAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAJ JAAAAAMAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAAAAAAAAAAAAAAAAJ 1 AA AA MA AAAA I ~ All. MAA AAAAAM ~ I AAA AAAAAAAAAAAAAAMA J J AAA A MAMAAMAA I J AAA AA I I AAAA MA I J AAAAAAAAAAAAAAA I I AAAAAAAAAA I J J I I I J 5 5 I j

I J 1------l------iZ-------3-----~--------S------6-----7-----1

Figure 28a

A sloppily writteD IDput. A couple of slurred dotted ei&htb .. sixteeDtb Dote pairs. Tbe secoDd sixteeDtb Dote bas a Ibarp accideDtal to Its left. Tbe secoDd tail OD eacb sixteentb Dote coincides witb a staff line; the visual cue sbowin& that it is • sixteentb Dote Ii tberefore abliterated.

51

1---------.---------2---------3---------4---------$---------6---------7---------1

I • IDDD I 1 +.+++.++++++.+.... I I .+++++++ I I + ••• +++ I I II I I 1 J J , 1 I I I I I I

I I 8 2~~~~~~~~~~~~~22~~~2222~22~~~~~22~22~1 1~2~~~22~~2~22~22222~22~222~2222~~2~22~222222222~222222222222222~2222222222~ DD I I a a I I I I I I J R I II I 2 .v9999 .9999999999999999999999999999999999999999999999999~ 199999999999999999999999999999999"'99 •• 9999'9999999999999999999999999999999~99 I

- J a • D J I I I I I I J 80 +.+++++ •• ++++.+++++++ •••••••••••• + ... . 1 .............. + •• ++.+++ •• +.+ ••• ++ •• ++++++.+++++++.+ •• +++++++.++++.++.+ ......... , 188 a • I 1 I I 3 ID 3 I II. 11$... ••• I I ••••••••••••••••••••••••••••••••••••••••••••••••••••• • ••• ••••• •• • •••••••••••••• 1 188888 10 IB 8B • aBBB • I J I I I I 1 8 I I I IDD8BBEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEFEEEEEEEEEEEEEEEI IEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE£EEEJ 1 10 I • I 4 4 I IDDDIDBDB J I MAAAMAMA I I J J an n I I ,....."''''tttt1rIM .. ,rttft'tf1Mr1 I I BUUUUUDBBH I 1 I J I J I ~ 5 • I 1 I 1------1------2---------3----4-----$------6------7-----J

Figure 28b

Horizontal liael .etected la tbe Imale

52

J---------I--------::!--------3---------4---&------6-----7-------. • I • I I I I I I AAA I • AA AAAA I I AA AAA I I M A~ I J A AAA I I A AM A I I AM AA AA I J MM A AA J I AAM A A Afl.fl.AA AA I J AAAM A A AMAA AA I I AAAAA A "'-AAAA M I I "'-AAA M A AAAAA AAA I J AAAA MA A AAMM A AAAAA I J AAA A AA AAA "'-A A AAMA I 2 AAA AAAAA MAAA AAAAAA AAAAAAA ::! I AAA AAAAA MAAA MAAM AAAAAM I I AA "'-AAAA AAAAA AA AI\ AIIAA I I AA AAAAA AAAAA MAAA MA I I A AAAAAA AAAA AAAAA AA I I A AAAAA A AA AA AA I I A AI\AAA A MAAA AAA J I A AAAAA A AAAM AM I I A AA A AAA I '. A AA A AAAA I 3 All. AA A AAAA 3 I Alit AA AA AMA • I AAAA A... AA MAA I I ItA AA All. A I 1 AlIt It... AA I 1 /ItA AAA ItA I I MA AAA A I J MA AAA A M AAA I I MA MA A ItA MA I I AA A AA AAA J 41 AA MAA AAAAAAA 41 I AAA MA AAAAA I J AAA A A A I J AAA AA J J All. AA I I A I I I I I J I I I , , • • J I 1-------1-------2--------3-------4--------'-----6------7-----1

Figure 28c

HorizoDtal .tart IIDes removed from tbe Im8lt.

53

1--------1-------:1--------3---------4-----1----6--------7------1 I ) I I I I I I I J I A • I AAA I I A AA J J A MA I I A M I t A J J It A I I A It J I A A A A J I A A A It A I 1 A It A A I I A A A A J I A AlIt It AA A A J J A A A itA It It 1 :I A A A AA A A :I I A A A A MA J 1 A A AlIt AA A J J A A A AA A J I A A A A A J I A A A AA A I I A A A AA A I I A A A AA A I I A A A A I J A A A A J 3 A A A A 3 I A A A A J I It A A A I I A A A 1 I A "A J J A A It J I A A A J I A A It I I A A A A A I 1 A A A I 4 A A A .. I It A AAM J I A A A I I A A J J 1\ A I J A J I I I ) I J I I ~ ~

• • I. I 1------I-------;!--------3----4-;.----a-----6-------7-----)

Figure 28d

The fiDal tbiDDed result. Tbe opeD Dolebeads on the second dotted eilbtb-sixteentb will be easily eliminated by the hilb level expert system. Deter-mininl tbat tbe second Dote or eacb couple is a sixteenth Dote, however, can only come from contextual Inference and backlbround knowledle.

Chapter 4

VI sua) Expert System

AFter low-level processing has extracted the

primitive Features From a piece of music, this

inFormation is passed to a visual expert system. The

expert system, written in PROLOG, uses knowledge about

musical notation to identiFy each of the Figures in the

score based on the primitives and the spatial

relationships between them. These primitives are the

lines, holes and blobs that are detected by the low-level

system. These Features are made avai1able to the visual

expert through several Files which are loaded by PROLOG.

For each primitive, there is information about its type,

its location and its orientation. The goal of the system

is to determine pitch and duration For each Figure in the

image. A "figure" is a collection of primitives that have

some relationship to each other. Usually, these

primitives are connected. When pitch and duration have

been determined, the information 15 output in a musical

representation language called NOTES (brieFly described

in Appendix D).

The visual expert's problem is to classiFy the

54

55

various Figures in handwritten music in a manner

consistent with the rules of musical notation. This has

been reFerred to as a "consistent labelling problem", and

tree-searching methods have been proposed to solve these

problems [Haralfck and Elliott, 1980]. Music presents a

complex domafn and a very large search space. Standard

search

bl ind

techniques,

and may

even with clever speed-ups~ are still

spend time investigating unlfkely

possibilities. Use of Frames provides a means or limiting

the search space and directing the search to produce a

more Focused problem solution.

ENGRAVE bu i 1 ds a hierarchy or rrames that

corresponds to the structure of musical notation. Lines

contain measures and measures contain notes. Slots of

higher level Frames in the hierarchy may contain the

names of lower level Frames. For example, line Frames

have a list of measures as one of their slots.

Communication among levels of the hierarchy is

achieved through production rules. These rules are

"attached" to the slots or the rrames and to the Frames

themselves. One or more "IF-NEEDED" rules are associated

with each slot and are invoked when a value is needed For

that slot. When an IF-NEEDED rule succeeds, the value

56

that was computed is added to the slot. "TO-FILL" rules

are associated with a frame and contain the slots that

are required to be filled for a frame. TO-FILL rules

automatically fire when a frame is created. Both TO-FILL

and IF-NEEDED rules are triggered "on demand." No values

are computed until they are needed. Once they are

computed, they are stored in slots so that they need not

be computed again. The visual expert is patterned after

CENTAUR [Aikins, 1980], a lung disease expert that uses

frames with attached rules to make diagnoses about lung

diseases based on data from tests.

Each measure frame must satisfy a constraint that

the sum. of the durations of the notes in 'a measure must

add to the number of beats in the measure. If this

constraint is not satisfied, the system will backtrack

and try new rules for various figures. Each time the

constraint Is checked for a measure, an evaluation

function is run. If the sum of the note durations for

that measure is equal to the number of beats in the

measure, then the evaluation function returns zero.

otherwise, the evaluation function returns a value that

indicates the degree that the constraint is violated.

When a set of notes is suggested that does not satisfy

57

the constraint, ENGRAVE wi 1 1 backtrack until the

constraint is satisFied or until all solutions for the

measure have been exhausted. IF no solution is Found that

completely satisFies the constraint, the solution with

the lowest value returned From the evaluation Function

(the solution that is "closest" to being correct) is used

for the measure.

The system provides the capabi 1 ity for the

application of constraints to any frame. These

constraints' are checked when the TO-FILL rule for a frame

has comp 1 eted runn f ng. ·Wh f 1 e ENGRAVE on 1 y conta i ns one

constraint, other constraints may be added to measures

and to other objects as the knowledge base grows.

4.1 ImplementatIon

The Frames and production rules in this system are

implemented in PROLOG using the techniques described in

[Chester, 1981]. Production rules are all of the form:

( (RULE ••• args ••• ) IF (condition-i) (conditfon-2)

(conditfon-m) (act 10n-l)

58

(action-m)

To retrieve values From a Frame slot, we use a Prolog

clause called RETRIEVE. RETRIEVE First to get the value

directly From the slot. IF the slot does not contain a

value, then RETRIEVE calls the IF-NEEDED rules that are

attached to the

the slot or iF

slot. IF there are no rules attached to

they Fail when they are called, then

RETRIEVE will return the deFault value For the slot. It

is expected that the IF-NEEDED rule will add the value it

computes to the slot.

The goal of the system is to Find the Figures in a

piece of music and to label them correctly using their

Features as inFormation. Frames play an important part in

this process because they speciFy the Features needed to

give a Figure a certain label. In addition, the presence

of a Frame suggests plausible combinations of Features

and thereby provides a Focus For program execution. The

program

Features.

begins

IF

associated with

possible- objects

by checking For important distinguishing

one or more of these Features is Found

a Figure, this suggests a class of

For the Figure and causes a frame to be

59

instantiated. Now the system begins to Fill out the other

slots in the Frame. Each Frame is assigned a label based

on the Features that are detected. In cases where there

is ambiguity or where a Frame is not completely Filled

out. there may be several possible labels For a frame.

When there is more than one possible label, these labels

are resolved using global constraints. Figure 29 shows a

Frame For a dotted eighth note.

4.2 An Example

This section shows the processing sequence of

ENGRAVE as it operates on a one-measure image. A dIagram

of the image is shown in Figure 30. The rules used in

this expert system are listed in Appendix A.

AFter loading

started simply by

the TO-FILL rule

the primitives file, the program is

creating a staFF Frame. This triggers

for staves which tries to fill the

"lines" slot and "measures" slot for the staFf. Since

neither slot has a value, the IF-NEEDED rules for these

slots will fire.

The rule that fills the "lines" slot of a staff

(note G45 notehead GI2) (note G45 stem GIS) (note G45 flags (G22) ) (note G45 dots (GI3) )

(note G45 duration 2.666)

(note G45 pitch (C S) )

60

blob for a notehead stem connected to notehead one flag connected to stem one dot fn the vicinity of and following the notehead Durations are represented by the denominator of the note-type, in this case, a 1/8 note has been modified by the dot.

(blob G12 SO 103 23) ; notehead (curve GIS «51 106)(52 106)(52 107)(52 108) .•• (51 122»)

; stem (curve G22 «52 122) ••. (40 130»)

Figure 29 Example of a note frame

;flag

61

E Q)

-&J (/)

> (/)

+J L-Q) Q X Q)

ttJ :J (/)

> 0 (T) Q)

J: Q) +J L-:J L. Oi 0 - "-

I.L.. co

+J (tJ "0

+J :J Q C

"-0

E CO L-0: q)

0

62

looks ror rive starr lines that are adjacent (no starr

lines between them). Starr lines are distfnguished during

low level processing rrom other horizontal lines by the

ract that staFr lines are very much longer. In this

example, there are only Five starf lines in the image, so

"finding them is trivial. In a larger image, ENGRAVE would

start with the Five highest staFr lines in the image.

When we say that a rule "looks "for" a starF's lines,

we mean that it simply perrorms a RETRIEVE operation.

This is how "computation on demand" works. When the value

or a slot fs needed, it is simply computed. I"f the value

has been computed, it is simply returned. IF no value has

been computed, the IF-NEEDED rule ror the slot is

triggered and computes the value. In our example, the

TO-FILL rule attempts to retrieve the value of the lines

slot. Since there is no value For the slot, the IF-NEEDED

rule fires and makes a conclusion about which primitives

are the starf's lines.

The TO-FILL rule for staves also looks for the

measures that are part of the staff. The IF-NEEDED rule

for measures repeatedly creates a measure frame until it

can Find no more measures in the starf. Each time a

63

measure Frame 15 created y the TO-FILL rule For measures

is run. This rule looks For a measure's bar lines and its

notes and then checks the notes against a constraint. In

our example, ENGRAVE labels the primitives 1 and 25 as

bar lines.

The First primitive that is Found is the hole

numbered 13. The hypothesis is made that this primitive

is the notehead of the current note. The "notehead" slot

in the note frame now contains a reFerence to h01e 13.

The system knows there are only a few kinds of musical

Figures that have open noteheads: whole notes y half

notes y sharps and naturals. The First primitive that the

TO-FILL rule looks For is a stem. IF there is a stem,

then we know that the note is a halF note. Otherwise, the

system must go on and try another rule. In this case, the

note does have a stem: the Figure numbered 16. The system

needs only to determfne whether there are any dots

modiFying the duration of the note. None are Found and

the IF-NEEDED rule for dots Fails and ENGRAVE must

include that this note is a halF note with the usual two

beat duration.

The next primitive that ENGRAVE Finds is a blob (3).

64

This is most likely a note with a closed notehead (such

as a quarter note or an eighth note) or a whole or half

rest. The TO-FILL rule. iF it Finds a blob that has not

already

and then

IF-NEEDED

notehead,

been labelled, assigns the blob as the notehead

retrieves the note's duration. This triggers an

rule for note duration. If a note has a closed

ENGRAVE looks for a stem; The notehead numbered

3 does have a stem so we know that the note is not a

rest. At this point, the system has to look for flags and

or dots to determine the duration of the note. The

IF-NEEDED rule For Flags fails because there are no flags

attached to the stem. The rule for dots concludes that

there is a dot (7). Therefore, ENGRAVE concludes that

this note is a dotted quarter note (1-1/2 beats).

The blob numbered 8 is the primitive used for the

next note. The reasoning process proceeds exactJy as

above until ENGRAVE looks for flags. This note has a flag

(15) attached to its stem. From the figure, it may not

seem that 15 and the note's stem are connected but

ENGRAVE has knowledge to account for the fact that some

of the connecting pixels were included as part of the

stafF line. There is a rule that understands that these

two features are connected because they both are

65

connected to a starr line and they are placed in a

relationship that suggests a stem and a rlag.

None or the remaining primitives in the image have

the correct relationships to each other to be labelled as

musical rigures. (One exception is the tie formed by 10

and 11 no knowledge of ties is currently present in

the system.) Each or these rigures receives a NIL label

indicating that no interpretation has been assigned.

At this point the notes are checked against a

constraint. The constraint requires that the sum of the

durations

or beats

time (a

durations

contains

of the notes in a measure must equal the number

in the measure. Since this measure is in 4/4

fact specified by the user), the sum of the note

must add up to four beats. This measure

a dotted quarter note (1-1/2 beats), an eighth

note (1/2 beat). and a half note (2 beats). The sum or

these durations is indeed four beats so the constraint is

satisfied. At this point all or the notes in the current

measure are known and the TO-FILL rule for the measure is

satisried.

there are

The rule for the staff is also satisfied since

no more measures in this image. On a larger

image, the TO-FILL rule ror a staff would create a new

66

measure 'frame and proceed to process the next measure.

The 'final conclusions in this example are summarized

in Table 1. ENGRAVE has built a smal1 'frame structure

showing a line that contains a single measure. The

measure contains three notes. The 'first note is a dotted

quarter note (1-1/2 beats) with a pitch oT "Aft. The

second is an eighth note "E" and the third is a hal'f note

"En. Figure 31 shows a messier measure. ENGRAVE's

conclusions are summarized in Table 2.

4.3 Conclusion

ENGRAVE can understand most conventional notation iT

it is reasonably legible. Even when the handwriting is

somewhat sloppy, as in Figure 31, ENGRAVE can understand

the notation. In Figure 31, ENGRAVE mistakes the natural

berore the second note Tor a sharp. This is due to the

'fact that the key signature is not known to the system.

ENGRAVE can tell when a horizontal line is being obscured

by a star'f line. In Figure 30, the system determines that

the second note fs an eighth note even though the 'flag is

closer to another note. Because ENGRAVE knows that star'f

lines obscure horizontal lines and because the system

67

Table 1 Summary of results from Figure 30

fDlnm .s.un valuers) Staff A Lines 20, 21, 22, 23, 24

Measures B

Measure B Staff A Bar-lines 1, 25 Notes C,D,E

Note C Measure B Dotted quarter note Notehead 3

Stem 2 Dot 7 Duration 1 1/2 beats Pitch A natural

Note D Measure B Eighth note Notehead 8

Stem 9 Flag 15 Duration 1/2 beat Pitch E natural

Note E Measure B Half note Notehead 13

Stem 16 Duration 2 beats Pitch E natural

.----------------------------Unaccounted for: 10, 11

· · · · r :

l"· I · · · · ·

68

~<

~

r ·f I

! J.

I · · I · · ~~~~~~~~~<~~~~~~~~~~<~«~ !

· · · · · · · • I

~-"""~H"""_H""H"""HH~_"HH""HH"~H __ "_hH_M~"_"""_""_~~

('I")

Q) s... :J 0)

La..

"'0 c to

+J rt)

'-4J "'0 C :J

0 +J

4J s... :J rt) (0 Q) E

+J -:J 0

~ ~

"0

Q) s... ~

C(

Table 2 Summary of Results from Figure 31

Er.aJru:. Sl21 Yahu~(&} ~ SJJn Yalue<&} Starr A Lines 1, 2, 3, 4, 5 Note E Measure B

Measures B Eighth note Notehead 14 Accidental note D

Measure B Staff A Stem 16 Bar lines 6. 35 Flag 17+18+20 Notes C,E,F,G Duration 1/2 beat

Pitch D sharp Note C Measure B Quarter rest Upper leg 7+8 Note F Measure B 0\

Lower leg 9 Eighth note Notehead 24 \0

Middle bar none Stem 23 Duration 1 beat Flag 17+18+20

Duration 1/2 beat Note D Measure B Pitch D sharp Sharp Center 11

Left leg 10 Note G Measure B Right leg 12 Half note Notehead 28, 29 Upper bar none Stem 30 Lower bar none Duration 2 beats Modifies note E Pitch D sharp

-----------------------------Unaccounted for: 19. 21, 22, 25, 26

10

knows the kinds of notes that should have Flags, ENGRAVE

is able to correctly determine the duration of both the

second and the third notes. ENGRAVE is also able to use

its knowledge to determine that the two curves at the

beginning of the measure in Figure 31 form a quarter rest

in spite of the Fact that the middle bar is obscured by

the staff line.

In the present implementation, the visual expert

system is separate From the low-level system. Once the

low-level system has run, it produces Files For the

visual expert and control is passed to the expert system.

As mentioned in Chapter 3, knowledge is used in the

low-level system as well as in the the visual expert

system. Ideally, these two levels should be integrated.

In this way, all of the knowledge in the vision system

could be available to every level of the system. In

addition,

backtrack

necessary_

First, a

when a high level rule Fails, the system could

into the lower levels of the system iF

Some work is needed to build such a system.

control structure is needed to handle

communication between the diFferent levels of the system.

Secondly, knowfedge has diFferent meaning at different

levels. A method must be determined For representing

71

knowledge at these different levels and for converting

between them.

Chapter 5

Spatial Planning For "usfcal Output

An important kind of planning involves the

management of objects in space and their interactions and

is called spatial planning. Spatial planning is a

resource allocation problem. It is diFFerent From the

usual robot planning in ArtiFicial Intelligence, however,

because it deals with geometric problems. Typically, A.I.

planners have dealt with dependencies or tasks and with

task sequencing, not with reasoning about geometry.

Spatial planning must also address the interactions of

objects in space and the methods needed to place them

properly.

Creating printed music is a spatial planning problem

characterized by a large solution space. To get an idea

of the potential size of the problem see Figure 3. Figure

3 shows a simple piece of music that has 139 notes and

50 measures. Each note has a wealth of inFormation

associated with ft including location, pitch, duration,

stem direction, and spacing inFormation. All of these

properties must be computed For every note, measure and

staFF so that each object is properly placed on a

72

73

sheet of paper. Ross gives over three hundred rules just

for beaming two eighth notes together [Ross, 1970, p.

104]. Because or the size or the space, some planning

must take place in order to avoid having to redo large

amounts or work. This planning activity must be guided by

knowledge or musical notation and by knowledge or how

musical rigures interact in space.

This chapter reports the investigation or spatial

planning ror printing music. The output stage or ENGRAVE

takes a computer representation of a piece of music and

generates a high qual ity output. ENGRAVE is a

hierarchical, spatial planner that uses a process called

"constraint guidance" to plan the spatial layout and

correct notation or a piece or music.

5.1 Plarvafng ahead

Berore producIng a manuscript, a copyist must plan

ahead to insure that a 11 or the measures fit on a page

and that . they 1 ine up properly on the stafr [Roemer,

1973] • Engravers plan by actually drawing sketches on

their plates to mark the positions or the notes [Ross,

1970]. Notice in Figure 3 that the last measure on each

74

line (other than the last) ends exact1y at the end of the

line. In addition, the notes have diFFerfng amounts of

space between them dependfng on their duration -- longer

notes take more space. To determine the placement of

notes on .a line so that they are properly spaced and

st i 11 Fi 1 1 the line, ENGRAVE plans ahead using a

technique called "constraint guidance"

ENGRAVE's equivalent to the engraver's rough sketch

is a hierarchical plan. The hierarchy represents the

hierarchical structure of music. At the highest level are

pages. To print a page, one must print the staFF lines

,that it contains. Likewise, to print a staFF line, its

measures must be printed. ENGRAVE computes the structure

of the plan and computes only the details that are

needed to make sure that the plan will work. Once this

inFormation is determined, all of the other parameters

needed to print the music can be calculated when the

music is actually printed. See Figure 32 For an example

of a plan the notatfon that it will eventually produce.

5.2 ConstraInt guIdance

Spatial planning can be viewed as a constraint

75

..... . -

<

76

satisraction process. A good indication or this can be

seen by noting the similarity between Gaschnig's [1974]

rormalization of constraint satisraction problems and

Eastman's [1973] characterization or the spatial planning

problem.

ENGRAVE uses constraints to guide its spatial

planning process. A constraint is derined as a relation

between plan variables [Stefik, 1981] or a requirement

that must hold true of the variables. In ENGRAVE, the

constraints are

figures' that, when

the required

satisfied,

relationships between

will result in correct

musical notation. We use the term "constraint guidance"

because the constraints are the goals of the system.

Because the constraints constitute goals, it can be said

that the constraints guide the spatial planning process.

Constraint guidance is a generate and test

procedure. (Sterik suggests that all planning is generate

and test.) ENGRAVE starts with an initial plan and tests

the plan to see whether it satisfies the constraints. If

any constraint is violated, a corrective action is taken

that modifies the plan, and the cycle is repeated untiJ

all constraints are satisfied.

71

Every constraint has a corrective action associated

with it that is taken if the constraint is not satisfied.

Each of these constraint-action pairs forms a

forward-chaining production rule with each having the

form IF (NOT constraint) THEN corrective-action. Using

this format, it is possible to build a knowledge base

about any domain that requires resource allocation.

Later, we will demonstrate this by showing that

constraint guidance can be used to format screens for

human-computer interaction.

5.3 Description

Execution of ENGRAVE proceeds in three stages:

initialization, constraint guidance and rendering. During

the initialization stage, a skeleton plan is built that

the constraint guidance stage uses. Measures are assigned

to staff lines at a rate of ten measures per line. The

constraint guidance stage uses this skeleton plan for the

constraint guidance process. When no more constraints are

violated, the rendering stage computes the values

necessary to dfsplay the music as indicated in the plan.

Constraint guidance and rendering are knowledge based

78

processes driven by rules.

There are really two kinds or rules in ENGRAVE. They

are called rendering rules and constraint guidance rules

arter the stage or the system that uses them. The main

dirrerence is that constraint guidance rules guide the

planning process and rendering rules compute the details

or a particular plan.

Constraint guidance rules are the constraint-

corrective action pairs mentioned above. These rules are

concerned with planning how to layout the piece or music

on a sheet of paper. They deal with the higher levels or

the plan hierarchy mainly starr lines and measures.

The constraint guidance rules are used by ENGRAVE listed

below:

1) Ir there exist two adjacent measures containing a whole measure rest or a multiple-measure rest, then combine them into a single measure containing a multiple measure rest.

2) Ir the last page in the sequence contains more than its maximum number or lines then create a new page following it.

3} Ir there exists a page with more than its maximum number of lines, then move the last line on the page to the next page.

4) If, for the last line in the piece, the sum or the minimum sizes of its measures exceeds the amount of space available for the line then

79

create a new line rollowing it.

5) Ir there exists a line such that the sum or the minimum sizes of its measures exceeds the amount or space available for that line then move the last measure down to the next line.

6) If there exists a 1 i ne, not the last 1 i ne, such that the sum or the sizes or its measures is less than the amount of space available on the 1 i ne then expand the measures uniformly to fill the 1 i ne.

7) If there exists a line such that the sum of the minimum sizes of its measures is less than or equal to the space available but the sum of the actual sizes is greater than the space available then reduce all of the measures uniformly so as to make the sum of their sizes equal to the space ava i 1 ab 1 e.

8) Ir there exists a measure whose size is less than its minimum size then expand the measure to its minimum size.

9) Ir there exists a beat-unit that is not full and there exists a flagged note that is adjacent to one of the notes in the beat-unit but is not a member of any beat-unit itselr then add the note to the beat-unit. (Note: A beat-unit is a group Or notes that are beamed together and treated as a unit. Typically, the notes in beat-unit will sum to one beat. By "flagged note" we mean that the note has duration less than a quarter note and will have a tailor a beam attached to its stem. )

10) If there exists a flagged note that is not a member Or any beat-unit then create a beat-unit with that note as its only element.

11) If there exist two adjacent sub-beat-units that are of the same level and are members or the same beat-unit then combine the two sub-beat-units.

80

These rules are sufficient to insure that. when the

constraint guidance process is finished. the plan places

all or the starr lines and allOT the measures correctly

and that all notes smaller than a quarter note are

grouped properly into beat-units.

Rendering rules are

stage of ENGRAVE. They

mainly used in the rendering

are used to compute properties

such as stem direction and note locations that can be

determined once a plan has been computed. Some oT these

ru1es are used during constraint guidance to compute

properties (such as

planning and which

Rendering rules are

frame slot, when

available.

5.4 An Example

measure size) which are needed Tor

may change as the plan changes.

invoked to compute a property. or

the value is needed but is not

To illustrate how the constraint guidance procedure

works. we will trace through a small example. The piece

of music is shown in Figure 8 as ENGRAVE produces ft.

First. the initialization section produces a

81

skeleton plan: ten measures to a line. Diagrammatically,

the skeleton plan is shown in Figure 33. Because the

initialization section blfndly assigns ten measures to

be necessary. This is each line, some

where constraint

adjustment

guidance comes in. Notice that the sum

or the minimum sizes or the measures in the first line is

210.75 spaces. (A space is the width Or a quarter note.)

Since lines are 160 spaces long on the paper being used,

the First line is much too Full. Ir we tried to print the

line with so many measures it would be completely

unreadable because there is not enough room on the line

for all or the notes in the measures. This is accounted

For by one of the constraints that ENGRAVE uses to plan

the layout or thfs piece or music. When constraint

guidance begins, the first rule that Fires is rule 5

which contains a constraint concerning the sum or the

minimum sizes of the measures in a line. When. the rule

fires. it moves measure 10 down to the beginning or the

next line. The new plan is shown in Figure 34.

BeFore we go on. we need to discuss brieFly the way

that a measure's minimum size is computed. Any two

adjacent figures in a piece of music must be at least

one space apart for the music to be legible.

line:

measure

minimum size:

1 2 3 4 5 6 7 8 9

16.2 25.2 34.65 45.3 31.8 10.2 26.4 10.2 4.2

Figure 33 Skeleton plan for Figure 8.

10 11

6.6 6.6

12 13

16.2 26.4

14

30.6

Q) N

line:

measure

minimum size:

1 2 3 4 5 6 7 8 9

16.2 25.2 34.65 45.3 31.8 10.2 26.4 10.2 4.2

Figure 34 Measure 10 has been moved

to the second line.

10 11

6.6 6.6

12 13

16.2 26.4

14

30.6

CD w

Add i tiona 1 1 y.,

duration oT

following it

84

the distance between notes varies with the

the notes. Each note has an amount oT space

that is a Tunction of its duration.

Consequently, the minimum size is determined from the

duration of the smallest note in the measure. Consider,

Tor example, the measure shown in Figure 35. The smallest

note in this measure is the first (a 32nd note) so the

distance between it and the next note can be no less than

one space. The third note is twice as long as the first

two but it does not, as we might suppose, get twice the

space. Instead, it gets one and one half times the space

oT the shorter note. As it turns out., music engravers do

not use a strict proportionality but tend to "skimp" on

their

this

space in order to fit more notes into a line. Using

"one-and-one-half" method, we can compute the

minimum amount oT space needed For a measure, adding one

space for the distance between the bar line and the Tirst

note.

Going back to the constraint guidance process and

Figure 34, we Find the sum of the min<imum sizes of the

measure in the First line still exceeds the amount of

space available For the line so constraint guidance rule

number 5 fires again moving measure 9 to the next line.

85

Figure 3S Notes with longer durations

take up more space.

Rule 5

moved to

36. Now

86

will continue to rire until measure 5 is rinally

the second line. The plan now looks like Figure

the second line is in violation or the line size

constraint. Since it is currently the last line in the

piece, rule 4 wi1l rire creating a new line with no

measures in ft. Because no measures were moved by rule 4,

the second line is still too rull and rule 5 rires moving

measure 14 down to line 3. Now, the line constraint is

satisried ror all lines and the plan looks like Figure

37.

At this point, we know that all Or the measures will

rit on their lines. However, the measures must also rit

exactly. That is, the bar line or the last measure must

line up exactly with the end or the starr. The constraint

guidance process continues.

Since it is desirable to have all or the measures in

a line be nearly the same size, they are initially

assigned actual sizes such that they are a1l the same

size and the sum or their sizes is the length or the

line. In the case Or line 2, each or the nine measure is

given 17.78 spaces. When we look at Table 3, however, we

see that some or the measures are smaller than their

line:

measure 1 2 3 4 5 6 7 8 9

minimum 16.2 25.2 34.65 45.3 31.8 size:

10.2 26.4 10.2 4.2

Figure 36 All of the measures in the

first line now fit on the line.

10 11

6.6 6.6

12 13

16.2 26.4

14

30.6

CD ....,

line:

measure 1 2 3 4

minimum 16.2 25.2 34.65 45.3 size:

5 6 7 8 9

31.8 10.2 26.4 10.2 4.2

Figure 37 The final configuration tbat ful1y

satisfies the line constraint.

10 11 12 13 14

6.6 6.6 16.2 26.4 30.8

T = 138.6

CD m

Table 3 Initially, an the measures on

a line are of equal size.

measure 5 6 7 8 9 10 11 12 13 minumum size 31.8 10.2 26.4 10.2 4.2 6.6 6.6 16.2 26.4 size 1 7 .78 1 7 .78 1 7 .78 1 7 .78 17 .78 1 7 .78 17 .78 1 7 .78 1 7 .78 T = 160

(]) '-'>

90

minimum sizes. This causes rule 8 to rire and adjust the

size Or measure 5. In Table 4. we now have measure 5

correctly sized but the sum or the measure sizes is

larger that the line size. This causes rule 7 to Fire and

all of the measure sizes are reduced.

To save some computation, only those measures that

are larger than their minimum size are considered

reducible. There are six reducible measures in this

example, so the amount of reduction is (174-160)/6=2.33.

IF any reduction would result in a measure being smaller

than its minimum size, the measure is leFt at that size.

The result or this reduction is shown in Table 5.

Because of measure 62. the sum of the measure sizes

is still too large. Rule 7 fires again. This time there

are Five reducible measures and the amount of reduction

fs (178 - 160)/5 = 3.6. When this reduction is applied to

the measure. we get the result shown in Table 6 and all

of the measures Fit properly on the line. This procedure

is applied to every line in the piece until all measures

are correctly sized. The only exception is when there are

only a few measures on the last line (as in this

example). In this case, no actions are taken and the

measure 5 minimum size 3 1.8 size 31.8

Table 4 Measure 5 expanded to its minimum size.

6 7 8 9 10 11 12 13 10.2 26.4 10.2 4.2 6.6 6.6 16.2 26.4

17.78 17.78 17.78 17.78 17.78 17.78 17.78 17.78

\0

T = 174

measure minimum size size

5 31.8 31.8

Table 5 An measures are reduced by the same

amount but not below their minimum size.

6 10.2

15.45

7 26.4 26.4

8 9 10 11 10.2 4.2 6.6 6.6

15.45 15.45 15.45 15.45

12 16.2 16.2

13 26.4 26.4

\D N

T = 178.05

measure minimum size size

5 31.8 31.8

Table 6 All measures are reduced again.

6 7 8 9 10 11 10.2 26.4 10.2 4.2 6.6 6.6

11.85 26.4 11.85 11.85 11.85 11.85

12 16.2 16.2

13 26.4 26.4 T = 160

\D w

94

measure size set to the minimum size.

5.5 Beet Unfts

Rules 9, 10, and 11 deal with groupings of notes

called "beat-units." A beat-unit is a group of notes

connected by a beam. See, for example, Figure 38. Each

note is an eighth note or a shorter note. The "sub-beat-

unit," a construct that was devised for ENGRAVE,

indicates grouping within a beat-unit. The beams on a

beat-unit indicate the sub-beat-units. In every beat-unit

there is a "major beam" connecting all of the notes in

the beat-unit. If any of the notes are shorter that an

eighth note, there are also "minor beams" to indicate the

duration of each note. This will become clearer as we go

through the example.

When

beat-units

ENGRAVE creates

are indicated in

its skeleton

the plan. A 11

plan, no

notes are

attached directly to their measures. The part of the plan

for Figure 38 is shown diagrammatically in Figure 39.

Each of the notes is a flagged note indicating that

its duration is that of an eighth note or less. Rule 10

95

Figure 38 A beat-unit.

line

notes 1 2 3 4 5

Figure 39 The same beat-unit as it

is represented in the skeleton plan.

6 beat units

\D 0'1

97

will create beat-units or sub-beat-units to change the

plan to look like Figure 40. One level is added to the

plan For each Flag that a note would require iF it were

notated by itselF. Rule 9 will group the beat-units'

together to Form a single beat-unit as shown in Figure

41. Rule 11 combines adjacent sub-beat-units of the same

level. When we combine at sUb-beat-unit levell, we get

Figure 42. All of the notes at level 1 are adjacent, so

they get combined into a single sub-beat-unit. Combining

at level

beaming

will get

another

1, which

2, we get Figure 43. The plan now contains the

structure of our example. The beat-unit itselF

one beam. Underneath the major beam will be

beam, corresponding to the sub-beat-unit at level

also spans the entire beat-unit. Below the

second beam there are two beams connecting notes 1 and 2

and connecting notes 4 and 5. These beams correspond to

the sub-beat-units at level 2.

5.6 RenderIng

Once all of the constraints have been satisFied, the

constraint guidance process is Finished and the rendering

phase begins. During thfs phase, all of the properties

are computed that are needed to print the layout as

measure

beat units

sub lui 1

sub lui 2 \0 en

notes 2 3 4 5 8

Figure 40 Beat·units and sub-beat·units created.

measure

beat units

sUb-beat-unit lui 1

sub lui 2

notes

1 2 3

Figure 41 Beat-units combined.

4 5 6

\D \D

measure

beat units

sub lui 1

lui 2

notes 1 2 3 4 5 6

-o o

Figure 42 Combining level-l sub-beat-units.

101

N

! .~ c::

::::a ::::a CI) - N

CI)

CU CD CD as -E .! :2 :2 0

c::

102

planned during constraint guidance.

Since the nodes in a plan represent objects, it is

natural For them to be represented as Frames. rendering

rules are "iF-needed" rules that are attached to slots

and that Fire when a slot is accessed which has no value.

In this way, all slot values are computed on demand. As a

result, the program needs only to display each object in

the the plan. As each object is accessed all of the

values needed to display the object are computed iF they

have not already been computed.

When a key signature or a time signature must be

displayed, there are rendering rules that determine the

between them. These rules are used proper spacing

whenever a change of key, time, or cleF occurs in the

music and at the beginning of the piece. Figure 44 shows

a diagram of proper spacing between each of these kinds

of figures.

Slot

propertfes

but that

values that are computed by rendering rules are

that are not essentfal to the planning process

are needed to display the figure. Some of these

properties are stem length, pitch, and key signature. A

103

......--o-~

Figure 44 Conventions for spacing.

104

list of the rendering rules can be round in Appendix A.

ENGRAVE is capable or producing most conventional

music notation. Figures 45 through 47 show some or its

capabil ities. An evaluation or other systems that produce

musical notation can be round in Chapter 2. We reel that

ENGRAVE compares ravorably with these systems and that it

has an added advantage because or the flexible nature of

its representation. ENGRAVE does not currently output

multi-line parts or music ror voice as these capabilities

require new rulesets. Appendix E contains the details or

the ENGRAVE's graphics including the methods used to

compute ties, slurs and beam slants.

5. 7 Constra' nt Gu f dance f n a Another Dana in

Constraint guidance can be applied to any domain

that involves the placement or objects in space. To

illustrate its use, we will present a second example of

constraint guidance as applied to the formatting or

graphics screens For human-computer interaction. We wi11

use the afrcraft carrier controller's screen discussed in

[Reilly and Roach, 1984]. Instead of two screens, we will

assume that we have one large screen on which we place

107

~

~ ~

~ •

, ~ t-

n ~ cG

c rL ·s

«I

!- .8 I- "C c b as

t WJ -~

·c r-- ::s ~.:... as u u ... .0 ::s

• tIOtaO .- C ~ --il

0 ..c

WJ

u Q. .. e "" l< U

I c <

I

~

,

108

all or the objects. These objects are: a plan view or the

aircrart carrier showing the rlight path and the

locations or the planes,

displaying the aircraFt

horizontal perspective or

a vertical perspective Tor

altitudes (the stack), a

the rinal landing phase (the

glide path), the marshall status board containing textual

inFormation about the aircrart, and a message area ror

system messages and user input. We want to place each or

these objects on the screen while rollowing good visual

design technIques so that no inrormation is obscured.

As in music, we start with a skeleton plan. In this

simpliFied version or the domain, two plans will seldom

be very dirrerent. The major blocks will be ordered in

the plan according to their relative importance as

speciried by the user. A three-level hierarchy exists in

this example because each block on the screen is composed

or several parts. Figure 48 shows a typIcal plan.

Initially, the nodes in the plan will not have location

values. The constraint guidance system will compute and

manipulate these location and size.

Constraints ror this example involve the concepts or

109

'i c: ..!. u n... f u

Cf.I

Cf.I .... ~ -0 ... -c 0 u

00 ... u

~ .-... u :a ... u ::I .!!l~ '" as ... u ... . ;

{; ... 0 ~

-¥s -1 ~ / .. '- /

~/

fa -a <

/

110

visual design and the requirement that inFormation must

not be obscured. We will use the following set of rules:

1) IF the placed, screen.

highest priority block then place it in the

has not been center of the

2) If the stack has not been placed, then place it to the leFt or right of the carrier.

3) IF the second highest priority block has not been placed, then place it so the Flight path points to it.

4) IF it

the glide path has not been placed, then place so that it points towards the aircraft

carrier.

5) If the marshall status board has not been placed, then place it in the largest free block of space.

6) If the message area has not been placed, then place it in the largest Free block or space.

1} IF two opposite sides of a block intersect an object or are off the screen, then scale the block down so as to clear at least one side.

8) If a side of a block is off the screen, then move the block away from that edge of the screen.

9) If a side of a block obscures another object, then move the block away from that object.

The blocks in this example are shown separately in

Figure 49. There are five planes in the flight path.

Three of these have their characteristics displayed on

the marshall status board and one is in the final landing

phase. Assume that the blocks are ranked in the following

111

o

o

MSB

Figure 49 The blocks found in the controller's screen.

1 12

order

stack.,

or importance:

gl ide path.,

plan view, marshall status board,

and message area. Rule 1 will place

the plan view in the center or the screen. Rule 2 places

the stack directly to the left of the carrier. The second

highest priority block, the marshall status board, is

placed by rule 3 below and to the right Or the plan view.

Rule 4 places the glide path above and to the right or

the plan view. Finally, the message area is placed in the

free area to the right of the screen by rule 6. The

result of these placements is shown in Figure 50. All of

the block have now been placed so rules 1 through 6 will

no longer trigger. Rules 7, 8 and 9 deal with the blocks

that overlap or

marshall status

the screen. A

go off the screen. Ru 1 e 8 wi I l' move the

board upward so that it is entirely on

plane is obscured however., by this move.

Before this is dealt with, rule 8 will trigger once more

to move the glide path onto the screen. The result is

shown in Figure 51. Now rule 9 triggers to move the

marshall status board to the right and uncover the fifth

plane. The glfde path still overlaps the plan view but no

information is obscured so the screen is lert as it is.

Figure 52 shows the final screen.

have given a simple example to show how

113

o

o

o

MSB

Figure SO Initial placement of objects.

114

o o

o

Figure 51 Resolution of objects off the screen.

!:J.

o

o

o

115

Figure 52 Final screen

116

constraint guidance can be applied to another domain. In

reality, the rules For visual design are not nearly as

well-derined as in music and they are not as rigid [Roach

et al., 1982; Roach et al., 1986]. A realistic system ror

visual design would have to capture this rlexibility and

the subjectivity of the domain. Spatial planning is

central to the realization or such a system.

5.8 Literature Review

Planning is an important part of problem solving.

Work on planning systems refers back to some of the

earliest work in Artificial Intelligence. GPS (Ernst and

Newell, 1969] and STRIPS [Fikes and Nilsson, 1971] are

especially notable. Both

essentially linear planners

level of detail. Multiple

or

and

levels

these systems were

could only plan at one

of abstraction were

provided by hierarchical planners, the first of which was

LAWALY [Siklossy, 1972; Siklossy & Dreussi, 1983]. Other

hierarchical planners are ABSTRIPS [Sacerdoti, 1974],

NOAH [Sacerdoti, 1977], and HOLGEN [Stefik, 1981]. It is

worth noting that HOLGEN used a technique called

"constraint posting" to'handle some or problems caused by

Interacting subgoals.

117

Constraint satisraction is also an important topic

in Artiricial Intelligence literature. In some domains.

it is desirable to derine constraints that limit the

search space or a system. This is especially true or

resource allocation problems [Wilkins, 1984]. ISIS [Fox,

1983], a

constraints

Job shop scheduling system, makes use or to build schedules ror machines and products.

As noted above, MOLGEN used constraints to reduce

sub-goal interaction in planning tasks. DEVISER [Vere,

1983] is a time-based planner that uses constraint

propagation to adjust start and end times ror tasks in a

plan. While TEX [Knuth, 1979] is not an AI system,

Knuth's "glue" is a kind or constraint.

Eastman's [1973] spatial planning system planned the

layout or rurniture in a room. The used a digitized

representation or space and a set or procedural

operators. Grason [1970] used a graph theoretic approach

ror arranging rooms in a building. In both cases, the

objects that are befngarranged had simple shapes and

could be arranged using a straightrorward state space

search. We reel that we have been able to build on this

roundation to allow for spatial planning using complex

118

objects.

The emphasis in this work has been on planning in

spatial domains rather than on musical output. There are

some musical output systems, however, that merit

attention. Perhaps the best known of these systems is

SMUT [Byrd, 1974]. This system can handle most

conventional music notation and can print multiple

voices. Other systems are mentioned in Chapter 2. All of

these systems use algorithmic methods to format their

output. We feel that much flexibility and power can be

gained by using heuristic methods from Artificial

Intelligence instead of algorithmic methods.

5.9 Conclusions

We have described a system that produces quality

musical output. The main thrust of this chapter however,

was not to describe how to produce correct musical output

but to describe constraint guidance by showing how it

operates in a musical domain. We have endeavored to do

this by discussing the principles behind constraint

guidance and by stepping through some examples.

119

There are some lfmitatfons to constraint guidance:

its dependence on an initial skeleton plan and the lack

of ability to relax constraints. Constraint guidance as

we have implemented it relies on an initial skeleton plan

to begin its operation .. This makes ENGRAVE a

generate-and-test system in a sense. One of the lessons

of DENDRAL [Lindsay et al., 1980, p. 148] is that a good

generate-and-test system must have a good generator. It

is important therefore, that a skeleton plan be a good

"first cut." This will reduce the amount of extra work

that the system must do. We could have had ENGRAVE

initially assign all of its measure to one line. Had we

done thfs, ENGRAVE would have spent a lot of time

creating new lines and moving individual measures to the

next line. Thfs could be a very serious performance

problem for a large piece of music. A system that uses

constraint guidance ought to have a good set of

heuristics for generating a skeleton plan.

It is

over-constrained

possfble that, fn some

situations could occur.

domains,

In our

implementation this would cause ENGRAVE to run forever.

In an over-constrained problem, there will always be a

violated constraint. It is desirable to be able to detect

120

over-constrained situations and intelligently relax

constraints to arrive at a best solution [Fox, 1983].

Barring this, the rule set must be designed such that

there is always a so1ution that satisfies all of the

constraints.

We believe that constraint guidance is applicable to

three dimensional domains. The chief requirement is that

objects can be decomposed into sub-parts. This allows Tor

the multiple levels of detail in a plan. It is not

required that the decomposition conform rigidly to a tree

structure but there must be a distinction between levels

of detail. A good example is a factory floor layout. Many

objects on a factory floor have parts that can be

configured in different ways. For example, a robot

station may have a controller that must be located near

the robot and a parts bin that must be within reach of

the robot arm. Additionally, machines may be combined

into larger entities such as an assembly line using

constraints about their location on the line. We believe

that this kind of spatial planning problem is a suitable

application for constraint guidance.

In summary, ENGRAVE is a spatial planning system

121

that uses a technique called constraint guidance to plan

the spatial layout of music. This technique is useful not

only ror musical layout, but for other spatial domains as

well. We believe that constraint guidance extends

existing spatial planning techniques to provide more

utility and power.

6.1 VIsIon

Chapter 6

Dfscussfon

The rirst section or ENGRAVE is a vision system that

can process handwritten musical notation. In the first

attempt to

general-purpose

reatures. These

expert system

these 'features.

it depended on

but 1 d the system, we tried using

image processing

reatures were

techniques

to be the

to detect

input to an

that used musical knowledge to understand

The expe'rt system worked qu i te we 1 1 but

an unreliable low-level system 'for its

input. Much time was spent changing parameter va1ues 'for

the low-level operators to get better results. However,

a set or parameters can only be tuned to work for a small

set of images and consistent results cannot be guaranteed

in general'. The solution to this prob1em was to use

domain knowledge at the low level. When this was tried,

there was

suggests an

be employed

successFu 1 •

a remarkable improvement in the results. This

important conclusion: domain knowledge must

at all levels ror a vision system to be

122

There are

consideration.

123

. several issues

The system as

that merit Further

it stands is still

separated into a low-level section and an expert system.

feel that these two leve"ls need to be integrated into a

single system. This would enable Full control over

knowledge at all levels. In this way, failure or a

high-level rule could result in backtracking into

low-level rules causing a reconsideration of pixel

relationships. When we build such an integrated system,

we will rind that several levels of knowledge exist.

Knowledge about image-processing, low-level

relationships, musical notation and control exist

simultaneously in the same system. This has been

discussed in [Clancey, 1982] and a Framework is proposed

to represent diFferent levels of knowledge and their

relationships. The use of Frames [Minsky, 1975] is

another method that is used to explicitly handle

diFferent levels of knowledge.

Another issue is the Flow of information between

levels. While ENGRAVE was being developed, it was found

that information From low-level processes was useFul at

the higher levels even though no conclusions could be

made earlier using the same inFormation. Low-1evel

124

inrormation, however, arter has a dirrerent meaning at

higher levels. The problem that needs to be addressed is

how inrormation is passed between levels and what forms

it should have to be most errectively used at any level.

6.2 GraphIcs

ENGRAVE uses spatial planning to produce its output.

This methodology is appropriate ror domains that involve

placement or objects in space. This work and previous

work have only dealt with two dimensional domains. An

interesting and useful erfort would be to extend these

ideas to three dimensions.

One of the worst shortcomings or the spatial planner

is speed. The system is slow partly because it is

implemented in an fnterpreted language and because it

must repeatedly search a large database. (There are

approximately 250 nodes in the plan for the full piece

shown in Chapter 4.) One or the 1essons of DENDRAL is

that the erriciency or the generator is important in

generate-and-test systems [Lindsay et al, 1980]. In our

case, this points to the corrective actions or the

production ru1es. What is needed for improvement is

corrective actions

Ideally,

125

that correct a situation more

when an unsatisFied constraint is completely.

detected, the

the constraint.

corresponding action should fully satisfy

Whether this is possible is an open

question, but there certainly is room for improvement.

6.3 Conclusion

The development of ENGRAVE provided a good vehicle

to study some interesting questions in artificial

intelligence, particularly expert systems. The use of

world knowledge for low-level vision computing and the

use of spatial planning for graphical output are the two

most important concepts that came out of this study.

While ENGRAVE, as it stands now, is not ready for

commercial use, it would require only a Few additions to

reach that point. Development of a music editor is the

largest effort that would be required. More effort is

also needed to make the system execute faster. However,

with the state of affairs in the music publishing

industry, a batch job that ran overnight still wou1d be

an economic improvement over current practices.

REFERENCES

J. S. Aikins, "Prototypes and Production Rules: A Knowledge Representation For Computer Consultations", Ph.D. dissertation, Deptartment of Computer Science, Stanford Univ. 1980.

C. Arcel11 and G. Samitl Di Baja, "A Thinning Algorithm Based on Prominence Detection," Pattern Recognition 13, no. 3, pp.225-235, 1981.

D. H. Ballard and C. M. Brown, Computer Vision, Englewood ClifFs, New Jersey: Prentice-Hall, 1982.

D. Byrd, "A System ror Music Printing by Computer," Computers and the Humanities, vol. 8, pp. 161-172, 1974.

D. Byrd, "An Integrated Computer Music Sortware System," Computer Music Journal 1, no. 2, pP. 55-60, 1977.

D. Chester, "Implementing Production Systems In Prolog," Internal Report, Deptartment or Computer and Inrormation Sciences, University Or Delaware, 1981.

W. J. Clancey, "The Epistemology of a Rule-Based Expert System: A Framework ror Explanation," Artiricial Intelligence, vol. 20, no. 3, pp. 215-251, 1983.

R. o. Duda and P. E. Hart, "Use or the Hough Transrorm to Detect Lines and Curves in Pictures," Communications of the AC M ., no. 1 5 , pp. 1 1 - 1 5 , 1 972 •

C. M. Eastman., "Automated Space Planning," ArtiFicial Intelligence 4, pp.41-64, 1973.

R. W. Ehrich, "A User's Guide to the EXTENDED CORE Graphics System ror the VAX 11/780," Internal Report, Department or Computer Science, VPI & SU, 1983 •.

G. W. Ernst and A. Newell, GPS: A Case Study in Generality and Problem Solving, New York: Academic Press, 1969.

I. D. Faux and M. J. Pratt, Computational Geometry For Design and ManuFacture, Chfchester,.West Sussex, England: Horwood, 1979.

126

127

R. E. Fikes and N. J. Nilsson. "STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving," ArtiFicial Intelligence 2, pp.189-208, 1971.

M. S. Fox, "Constraint-Directed Search: A Case Study of Job-Shop Schedu 1 I ng, 'f Ph. D. d i ssertat i on , C.omputer ~cfence Department, Carnegie-Mellon University, 1983.

J. Gashnfg, ttA Constraint SatisFaction Method For InFerence Making," Proceedings of the 12th Annual Allerton ConFerence on Circuit and System Theory, University of Illinois, Urbana-Champaign, pp.866-874, 1974.

D. A. Gomberg., "A Computer-Oriented System Printing," Computers and the Humanities, pp.63-80, 1977.

For Music vo 1 • 1 1 ,

J. Grason, "A Dual Linear Graph Representation For Space-Filling LocatIon Problems of the Floor-Plan Type.," in Emerging Methods in Environmental Design and Planning, G. Moore (ed.), Cambridge, Mass.: The MIT Press, 1970.

R. M. Haralick, "A Measure For Circularity of Digital Figures," IEEE Transactions on Systems, Man and Cybernetics., vol SMC-4, pp. 394-396., 1974.

R. M. Haralick and G. L. Elliott, "Increasing Tree Search EFFiciency For Constraint SatisFaction Problems," ArtiFicial Intelligence 14, pp.263-313, 1980.

R. M. Haralick, S. Krusemark Introduction," Internal Report, Laboratory, VPI & SU, 1981.

and K. NeikIrk, "GIPSY: Spatial Data Analysis

L. A. Hiller, Jr. Printing," Journal

and R. of

A. Baker, "Automated Music Music Theory 9, no.l,

pp. 129-162, 1 965.

D. R. HoFstadter, Godel, Escher, Bach: An Eternal Golden Braid, New York: Basic Books, 1979.

B. W. Kernighan and P. J. Plauger, SoFtware Tools, Reading, Mass.: Addison-Wesley, 1976.

D. E. Knuth, TEX and METAFONT: New Directions in Typesetting, BedFord, Mass: Digital Press., 1979.

127

128

S. Krusemark and G. Elliott, Documentation," Internal Report, Laboratory, VPI & SU, 1981.

"RATFOR Spatial

Preprocessor Data Analysis

R. K. Lindsay, B. G. Buchanan, E. A. Fiegenbaum, J. Lederberg, Applications or Artiricial Intelligence ror Organic Chemistry: The Dendra) Project, New York: McGraw-Hill, 1980.

J • T • Maxwe 1 1 I I I and S. M. Ornstein, "Mockingbird: A Byte, Vol 9, no. 1, pp. 384-410, Composer's Ameneunsfs,"

1984.

L. G. Minor Segmentation or Transactions on 1 94 - 20 1, 1 98 1 •

and J. Sklansky, "The Detection Blobs in Inrrared Images," Systems, Man and Cybernetics,

and IEEE pp.

M. Minsky, "A Framework ror Representing Knowledge," in The Psychology or Computer Vision, ed. Winston, New York: McGraw-Hill, 1975.

J. A. Moorer, "On the Transcription or Musical Sound by Computer," Computer Music Journal 1:4, pP. 32-38, 1911.

A. M. Nazir Segmentation: An Pattern Analysis 555-577, 1984.

M. Piszczalski Transcriptions," 1977.

, and M. D. Levine, "Low Level Image

Expert System," IEEE Transactions on and Machine Intelligence, no. 6, pp.

and B. Computer

A. Galler, "Automatic Music Music Journal 1:4, Pp. 24-31,

D. S. Prerau, "Computer Pattern Recognition of Printed Mus i c , " Proceed f ngs of the Fa 1; 1 Jo i nt Computer ConFerence, AFIPS ConFerence Proceedings, 1971.

D. S. Prerau, "Do-Re-Mi: A Program that Recognizes Music Notation," Computers and the Humanities, vol. 9, pp. 25-29, 1975.

S. S. Reilly and J. W. Roach, "Improved Visual Design ror Graphics Display," IEEE Computer Graphics and Applications, vol. 4, No.2, pp. 42-51, 1984.

J. Roach, J. A. Pittman, S. Reilly and J. Savarese, "A Visual Design ConSUltant," International Conrerence on

129

Cybernetics and Society. Seattle. Wash •• 1982.

J. W. Roach and G. S. Fowler. "The HC Manual: Virginia Tech Prolog." Internal Report. Deptartment of' Computer Science. VPI & SU. 1983.

J. W. Roach. S. S. Re i 11 y and J. E. Tatem, "I nte 11 i gen't Tools: An Automated Advisor f'or Visual Design," in HumanComputer Dialog Design, eds. Ehrich and Wil leges, 1986, pp. 146 - 163.

C. Roads, "A Note on Music Printing by Computer," Computer Music Journal 5, no. 3. pp. 57-59, 1981.

C. Roemer, The Art of' Music Copying. Sherman Oaks, Calif'.: Roerick Music Co., 1973.

A. Rosenf'eld and E. Johnston, "Angle Detection on Digital Curves," IEEE Transactions on Computers, pp. 875-878, 1973.

T. Ross, The Art of' Music Engraving and Processing, Miami: Hansen Books, 1970.

E. A. Sacerdoti, "Planning in a Hierarchy of' Abstraction Spaces," Art if' i cia J I nte 1 1 1 gence 5, 1974, pp. 1 15 - 135.

E. A. Sacerdoti, A Structure f'or Plans and Behavior, New York: Elsevier Publishing, 1977.

SIGGRAPH, "Status Report Of the Graphics Standards Committee," Computer Graphics (13), August 1979.

L. Siklossy, "Modelled Exploration by Robot," Technical Report 1, Computer Sciences Dept., University of' Texas, Aust in, 1972.

L. Siklossy and J.Dreussi, "An EfFicient Robot Planner Which Generates Its Own Procedures." Proceedings of' the Third International Joint Conf'erence of Artif'fcia1 Intelligence, Stanf'ord, Ca1if., pp. 423-430, 1973 reprfnted in Tutorial on Robotics, eds. C. S. Lee, R. C. Gonzalez and K. S. Fu, 1983.

L. Smith, "Editing and Printing Music by Computer," Journal of Music Theory 17, no. 2, pp. 293-309, 1973.

M. Stef'ik, "Planning with Constraints (MOLGEN: Part 1),"

130

ArtiFicial Intelligence 16, no. 2, pp. 1-39, 1981.

K. Stone, Music Notation in the Twentieth Century, New York: Norton, 1980.

s. Tsuji and F. Matsumoto, "Detection of Ellipses by a ModiFied Hough TransFormation," IEEE Transactions on Computers, vol c-27, pp. 777-781, 1978.

s. A. Vere, "Planning In Time: Windows and Durations For Activities and Goals," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 5, pp. 246-267, 1983.

L. T. Watson, K. I. Arvind, R. W. Ehrich, and R. M. Haralick, "Extraction of Lines and Regions From Grey Tone Line Drawing Images," Pattern RecognItion 17, No.5, pp. 493-507, 1984.

D. E. Wilkins, "Domain-Independent Planning: Representation and Plan Generation," ArtiFicia1 Intelligence 22, #3, 1984, pp. 269-301.

Appendix A

V I .ue 1 Expert Au Ie.

This appendix and Appendix B contain listings or the

rules used by ENGRAVE. The most important product or an

expert system is the knowledge that has been codiried.

Thererore, these rules are listed in English so that

someone else may make use of the knowledge.

A. I Ru 1 es about note durat fOIls.

Note durations are expressed as the reciprocal or

the mathematical value or the note's duration. Hence the

duration or a quarter note would be expressed as 4. The

duration or a dotted quarter note (one and one-halr

beats) would be 2.6667 (or 8/3).

Ir a note has a closed notehead, then its duration

is computed using the the formula 2(F+D+2)/3D• Where

F is the number or rlags and D is the number or dots

associated with the note.

Ir a note has an open notehead and there ,are two

curves that intersect the notehead and one curve is to

131

132

the leFt or the notehead and the other curve is to the

right or the notehead and one or the curves rises higher

than the other then the note is a natural and its

duration is NIL.

If a note has an open notehead, its duration is

computed by the rormula 2oT/3o where T is the type or

note (2 ror a halF note, 1 For a whole note) and 0 is the

number of dots that modify the note.

If a note has no notehead and there are two curves

such that both intersect the middle line of the stafF and

one runs from the middle line highe~ and one runs rrom

the middle line lower and both nearly vertical then the

note is a quarter rest and has pitch R (for rests).

All other notes have duration NIL

A.2 Rules about measures

To compute

create a note

Then add the

measure.

the notes of a measure, recursively

frame until the create operation fails.

list of notes to the NOTES slot of the

133

Ir a measure has a previous measure, then the rirst

bar-line is the second bar-line or the previous measure.

IF a measure has no previous measure, then its First

barline is the lertmost curve that is nearly vertical and

fntersects all or the starr lines.

To compute the second bar-line or a measure, rind a

curve that is nearly vertical and intersects all or the

starr lines and has no similar curve between it and the

rirst barline.

IF a measure is in the top line on the page, then

the top boundary or the area covered by the measure is

the top or the image.

IF a measure has another starr line above then its

top boundary is halr the distance between the top line of

its starr and the bottom line or the starr above it.

Ir a measure is in the bottom line or the page, the

measure's bottom boundary is the bottom Or the image.

134

Ir a measure has another starr line below it then

its bottom boundary is halr the distance between bottom

line or its starr and the top line or the starr below it.

The lert boundary or a measure is the co1umn

coordinate or the measure's rirst bar line.

The right boundary of a measure is the column

coordinate or the measure's second bar line.

A.3 Ru I as about notes

A.3.1 TO-FILL rules 'For notes

Ir there is an unused hole in the image. then treat

it as a notehead and retrieve the note's duration and

pitch.

Ir there is an unused blob in the image. then treat

it as a notehead and retrieve the note's duration and

pitch.

Ir there is an unused curve in the image. then

assign it to a note and retrieve the note's duration and

135

pitch.

A.3.2 IF-NEEDED ru1es For notes.

Ir a note has an open or a closed notehead and there

is a set or connected near1y vertical curves one or which

intersects the notehead, then conclude that these curves

are the note's stem.

Ir a note has a closed notehead and the note has a

stem and there is a curve that intersects the stem on the

opposite end rrom the notehead, then add the curve to the

list or rlags ror the note.

Ir a note has a closed notehead and the note has a

stem and there is curve that. intersects a starr at one

end and the note's stem fntersects that same starr line

and there are no curves between the stem and the curve,

then add the curve to the note's list of Flags.

IF a note has a closed notehead and the note has a

downward pointing stem and there is a curve that is below

and to the right or the note's stem and there are no

curves between the stem and the curve being considered,

136

then add the curve to the note's list or rlags. (This

allows ror stems that are drawn too short. One end or the

rlag should be almost directly below the stem.)

Ir a note has a closed notehead and the note has an

upward pointing stem and there is a curve that is above

and to the lert or the note's stem and there are no

curves between the stem and the curve being considered,

then add the curve to the note's list or rlags.

A note's starr is the starr that contains the

measure that contains the note. (This one is a litt1e

obvious but it is a necessary piece or knowledge.)

Ir a note has a stem and the end or the stem ,

opposite its intersection with the notehead, then the

note has a downward pointing stem.

Ir a note has a stem and the end or the stem

opposite its intersection with the notehead, then the


Ir a note has a stem which does not intersect the

notehead and the stem is above the notehead, then the

137

note has an upward pointing stem.

IF a note has stem which does not intersect the

notehead and the stem is below the notehead, then the


Note: In this implementation, a note may have only one

dot or none.

IF a note has a downward pointing stem and there is

a dot in the measure which is to the right and above the

notehead, then add this dot to the list of dots For this

note.

IF

dot in

notehead,

note.

IF

a note has an upward pOinting stem and there is a

the measure which is to the right and below the

then add this dot to the Jist of dots For this

there is a dot in the same measure as a note. and

the dot is directly to the right of the notehead and

there are no other noteheads between the dot and the

notehead, then add this dot to the list of dots For this

note.

138

A.4 Rul.s about stav.s.

To rill a stave rrame rind its lines and rind its

measures.

The lines or a starr consist or rive "starr-lines"

that are equal dfstances apart and that have no other

starr lines between them.

The lert end or a starr is the average or the lert

end coordinates or all the starf lines.

The right end of a staff is the average of the right

end coordinates of all the starr lines.

To rind the measures or a starf, create a measure

rrame repeatedly until the TO-FILL rule rails.

Appendix B

SpatIal Planner Rule.

B.l Constraint GuIdance Ru1es

1) Ir there exist two measures such that: - they are adjacent; and - they both contain only a whole rest;

then combine them into a single multiplemeasure rest.

2) IF there exists a line such that: - it is the last line; and - the sum or the minimum sizes of the

measures is greater than the line size; then create a new line arter this one.

3) Ir there exists a line such that: - it is not the last line; and - the sum of the minimum sizes or the

measures is greater than the line size; then move the last measure or the line down to the next line.

4) Ir there exists a line such that: - it is not the last line; and - the sum or the minimum sizes or the

measures is less than the line size; then expand all the measures in the line by (L-S)/n where L is the 1 ine size, 5 in the sum or the measures mlnlmum sizes, and n is the number or measures in the line.

5) IF there exists a line such that: - the sum or the measure sizes is greater

than the line size; and - the sum or the minimum sizes is 1ess than

or equal to the line size; then Find the least complex measure in the line with (size > min) and reduce it by min(sum_sizes - linesfze, size - min).

6) IF there exists a measure such that its size is less than its minimum size then expand the measure by (min - size).

139

140

1) IF there exists a beat-unit such that it is not Full and there exists a note such that:

- it is a Flagged note; and it is adjacent to one of the notes in the beat-unit; and

- it is not in another measure or beat-untt; then add that note to the beat-unit.

9) IF there exists a note such that: - it is Flagged; and - it is not already in a beat-unit;

then create a beat-unit with that note as its only element.

10) IF there exist two sub-beat-units such that: - they are the same level; and - they are adjacent;

then they can be combined.

B.2 Rendering Rules

IF the pitch language), the ( I mp 1 i ed tie)

IF the NOTES is not a rest,

IF the NOTES and the note notehead.

of a pitch

duration then the

note is not deFined (in the NOTES is the same as the note beFore it.

value For a note is 1 and the note note is a whole note.

duration value For a note starts with a 2 is not a rest, then the note has an open

IF the NOTES duration value For a note starts with 4 or greater and the note is not a rest, then the note has a closed notehead (as in quarter notes).

IF a note is a rest and has a duration value of 1, then the note is a whole rest.

IF a note is a rest and has a duration value that starts with 2, then the note is a halF rest.

IF a note is a rest and has a duration value of 4, then the note is a quarter rest.

141

IF a note is greater., then sma 11 er) •

a rest the note

and has a duration va1ue of 8 or is a eighth note rest (or

The vertical coordinate For rests is the same as the center line of the staFF.

Notes in the treble cleF are oFfset vertically from the center of the staFf by

(3.5 * a * S) - 17 spaces

where a is the octave number For the note and 5 is the oFfset From C for the letter name of the note.

Notes in the bass cleF are offset vertically from the center of the staff by

The 2.5.

The

The

The

The

The

The

(3.5 * a * 5) - 11 spaces

where a is the octave number for the note and 5 is the ofFset from C For the letter name of the note.

sequence number (distance in spaces from C) for A is

sequence number For B is 3.

sequence number for C is o. sequence number for 0 is 0.5.

sequence number for E is 1 •

sequence number for F is 1 .5.

sequence number for G is 2.

Full measure rests (whole rests) are placed in the center or the measure.

The First note in a measure always starts one unftsize from the start of the measure.

IF a measure contains a front repeat (leFt-handed)., then the first note in a measure is placed 3 spaces away from the bar 1 fne.

142

IF a measure is the First in its line and the line contains a time signature at its beginning, then the rirst note in the measure is placed 2-1/4 spaces aFter the time signature.

IF a measure is the rirst in a line, and the line contains a key signature at its beginning, then the First note in the measure is placed 2-1/2 spaces past the last accidental in the key signature.

IF a measure is the First in its line, and the line contains no key or time signatures at its beginning, the First note in the measure is placed 1-1/2 spaces From the beginning of the line.

IF a measure note in the signature.

has a time signature change, then the First measure is placed 2-1/4 spaces From the time

IF a measure has a key signature change, then the First note in the measure is placed 2-1/2 spaces From the last accidental in the key signature.

IF a measure has no time or signature changes, then the First note in the measure is placed 1-1/2 spaces From the First bar line of the measure.

IF a note is not the First note in horizontal coordinate depends on measure location or the last note as Follows:

where:

p * U * (5 - E) + L (M - E)

its measure, size and

P is the space allocated to the last note; and U is the "unltsize" of the measure; and 5 is the size of the measure; and

i'ts the

- E is the amount of s~ace needed In the measure For signature changes and repeats; and

- M is the minimum size of the measure.

IF a note has an accidental sign, the note should be moved 1 space to the right and the sign should be placed 1 space beFore the note.

143

The "space" property or a note (the amount or space the note gets within a measure) is computed by:

where:

5 .. T V

5 is the spacing value or the note from the measure's prorile; and T is the type or the note (4 ror quarter notes); and V is the reciprocal or of the notes duration in beats.

Note that this space is computed as ir the measure were at minimum size.

The stem direction ror notes in a beat unit is the same as the stem direction of the note in the beat unit that is rarthest rrom the center line of the stafr.

IF a note is part or a beat-unit, then its stem direction is the same as that or the beat-unit.

If a note's horizontal location is below the center line of the staFF, then its stem direction is "up".

Ir a note's horizontal coordinate is above or on the center line of the staFr, then its stem direction is "down".

Ir a note is part of a beat-unit, then its stem is long enough to intersect with the major beam of the beat-unit.

Ir a note is more than one ledger line outside of the staFr, then the stem is long enough to intersect the middle line of the starF.

IF a note has more than two flags, then the stem length is computed (in spaces) by:

F + 1.5 where F is the number or Flags.

In general, stems are one octave (3-1/2 spaces) long.

IF a note is not a whole note, and not a rest, then the note has a stem.

144

If a note is dotted and is on a line. then the first dot is 1/2 space above and 1-1/2 spaces to the right of the notehead.

If a note is dotted and is on a space, then the rirst dot has the same vertical coordinate and is 1-1/2 spaces to the right of the notehead.

Dots ror the same note are placed one space apart horizontally and all have the same vertical coordinate.

If a note is above or below the staff., then the number of ledger 1 i nes is computed by:

0-2

where 0 is the absolute value or the orrset or the note rrom the center 1 i ne or the sta"Fr in spaces.

The slope or a beat-unit is the linear regression slope or the line rormed by the noteheads in the beat-unit.

For the primary beam. the note closest to the middle line of the staff determines the location of the beam.

For sub-beat-units., the point on the beam is one line lower/higher than the primary beam for each level past 2.

Ir a note is the only element or a sub-beat-unit and is the last note or its beat-unit. then it has a "Fractional beam that points to the left.

If a note is the only element or a sub-beat-unit and is the first note of its beat-unit, then it has a fractional beam that points to the right.

Ir a note is the only element of a sub-beat-unit and is in an odd position in its beat-unit., then it has a fractional beam that pOints to the left.

If a note is the only element of a sub-beat-unit and is in an even position in its beat-unit. then it has a fractional beam that points to the right.

If a beat-unit or sub-beat-unit does not have a fractional beam., then the X-coordinates of the endpoints or its beam are the same as the X-coordinates or the stem ends of the first and last note in the unit.

145

IF a beat-unit has more than one note, then it has a beam.

IF a note has Flags, then the First Flag is located at the end of the stem and subsequent Flags are each located one space closer to the notehead.

IF a note has an accidental, it is located 1-1/4 spaces to the lert of the notehead and has the same V-coordinate.

The key signature of a line is the same as the key signature of its first measure.

If a line is the first in the piece, then print its key signature when the line is printed.

IF a line has a diFFerent key than the previous line, then we print its key signature.

No key signature is printed for lines other than the first unless the key has changed since the last line.

No key signature is printed iF the line is in the key of C.

The cler of a line is the same as the clef of its first measure.

The time signature of a line is the same as the time signature of its first measure.

IF a line is the first on its page, then its X position is

120 120 -------

n + 1

in spaces where n is the number or lines on the page.

A line's column coordinate is computed by

120 L -------

n + 1

in spaces where L is the location of the previous location and n is the.number of lines on the page.

IF a measure is the last in its line, the line is not the

146

last in the piece. and the next measure does not have a key change, then the measure's bar line is placed at the end or the line. (Note: the bar line that is associated with a measure is the one at the end or the measure.)

Ir a measure contains a rest ror its entire duration. and it is not the rirst measure or the line, then the location or the bar line is computed rrom the location of the previous bar line added to the size or the measure.

A bar line's horizontal coordinate depends on measure size and the location of the 1ast note in the measure as follows:

where:

p * u * (S - E) + L (M - E)

P is the space a1 located to the last note; and U is the "unitsize" or the measure; and 5 is the size or the measure: and E is the amount of space needed in the measure for signature changes and repeats; and M is the minimum size or the measure.

The vertical coordinate of a bar line is the same as the center line or the staff.

If a measure is the first in its line and has a starting repeat, then its starting point is the same as the horizontal coordinate or the repeat sign.

Ir a measure point is 3/4 signature.

is the first in its line, then its starting spaces past the last sign in the lines time

Otherwise. a measure's starting point is the same as position of the second bar line of the previous measure.

In a given measure. ir there exists a note such that: it is not the first note in the measure; and

- it has an accidental; and - its predecessor is of Jess than or equa1 duration; then the measure's unitsize is three times the size· of a horizonta1 space

otherwise it is twice the size of a horizontal space.

147

IF a measure is in the 1ast line on a page and the sum of the minimum sizes of the measures on the line takes up less than 1/3 of the space on the line. then set the size of the measure equal to the minimum size of the largest measure on the 1 ine.

The size of a measure is computed by the space available For a line divided by the number of measures in the line.

IF a measure is the First or last measure in the piece (These measures are oFten incomplete), then the size of the measure is computed by

where

o *. 5 * L

Num it N

o is the denominator of the time signature; and Num is the numerator of the time signature; and 5 is the sum of the durations of the notes in the measure (this is equal to the time signature For complete measures); and L is the amount of space in the 1ine; and N is the number of measures in the line.

IF a measure is the First in the piece, then its time signature is the same as the time signature of the piece.

IF a measure has a time signature change, then its time signature is the same as the new time signature.

IF a measure is not the First in the piece and does not have a time signature change in it, then its time signature is the same as that of the previous measure.

IF a measure is the First in the piece, then its key signature is the same as the key signature of the piece.

IF a measure has a key signature change, then its key Signature is the same as the new key signature.

IF a measure is not the First in the piece and does not have a key signature change in it, then its key signature Is the same as that of the previous measure.

IF a measure is the First in the piece, then its cleF is

148

the same as the clef of the piece.

If a measure has a clef change, then its clef is the same as the new clef.

If a measure is not the first in the piece and does not have a clef change in it, then its clef is the same as that of the previous measure.

If a clef is the clef at the beginning of a line, then it is located 2-1/2 spaces past the beginning of the line. Its vertical coordinate is the same as the center line of the staff.

If a clef is part of a measure (because of a cleF change), then it is located 2-1/2 spaces to the left of the previous bar 1 ine ..

If a key signature is part of a measure (a key change) then the horizontal position of its first sign is 1-1/2 spaces past the bar line.

IF a key signature is part of a line, then the horizontal location of the first sign is 2 spaces past the c1ef sign.

If a time signature is the in the same measure as a key signature, 'then the time Signature is positioned 2-1/4 spaces past the last sign of the key signature.

If a time signature is part of a line, then its horizontal is 3-1/4 spaces past the clef sign.

If a time signature is part of a measure and there is no key signature in the measure, then its horizontal position is 1-3/4 spaces past the bar line.

Note: In the following, "front repeat" re'fers to the starting repeat figure in a repeated section of music and "end repeat" reFers to the terminating repeat Figure.

If a 'front repeat is in the first measure of a line, then its horizontal position is 2-3/4 spaces past the time Signature.

If a 'front repeat is in the same signature change, then its horizontal spaces past the time signature.

measure as a time position is 2-3/4

149

IF a front repeat is in the same measure as a key change, but the measure has no time change, then the horizontal position of the repeat is 2-3/4 spaces past the last sign in the key signature.

If a front repeat is in a measure that is the first in its line, and there are no other figures at the start of 'the line (cleFs, etc.), then the horizontal location of the repeat is the same as the start of the staff line.

If a Front repeat is in measure that is not at the start of a line and the measure has no other key, time, or clef changes, then the horizontal position of the repeat is the same as that of the previous bar line.

If an change, sign.

end repeat occurs in a measure preceding a clef then locate the repeat 2-1/2 spaces before the clef

IF an end repeat occurs in a measure with no clef change following, then the horizonta1 location of the repeat is the same as the location of the measure's bar line.

If a line does not contain a repeat in the first measure, then the amount of space on the line for notes is lessened From the normal length of staff line by 6-1/2 spaces plus one space for each sign in the key signature.

I f ali ne has a new key signature, a new time signature', and a repeat in the first line, then the space on the line for notes is lessened from the normal length of a staff line by 10 spaces plus one space for each sign in the key signature.

If a line has a new key signature and a new time signature but no repeat in the first measure, then the space on the line for notes is lessened from the normal length of a stafF line by 6-1/2 spaces plus one space for each sign in the key signature.

If a line has a new key signature and a repeat in the first measure then the space on the line for notes is lessened from the normal length of a stafF line by 7-3/4 spaces plus one space For each sign in the key signature.

IF a line has a new time signature and has a repeat in the First measure then the space on the line For notes is lessened From the normal length of a starr line by 6

150

spaces.

IF a line has a new key signature only, then the space on the line For notes is lessened From the normal length of a staFF line by 4-1/2 spaces plus one space For each sign in the key signature.

IF a line has a new time signature only, then the space on the line For notes is lessened From the normal length. of a staFF tine by 2-1/2 spaces.

IF a line has a repeat in the First measure and no changes in key or time, then the space on the line For notes is lessened From the normal length or a staFF line by 1-1/2 spaces.

IF a line has no changes and thererore no special Figures to allow space ror then the space on the line ror notes is equal to the normal length or a stafr line.

IF a line has a diFFerent key signature than the line preceding it then both the clef and the key signature must be printed at the beginning of the line.

If a line has a diFferent clef than the line preceding it then both the cleF and the key signature must be printed at the beginning of the line.

IF a line has a different time signature that the line preceding it then the time signature and clef must be printed at the beginning of the line.

The mInImum size of a measure is determined by the following Formula:

where:

S * U + E

S is the sum of the durations of the notes in the measure; and

- U is the unitsize For the measure; and E is the amount of extra space needed for signatures, cleFs, etc.

'Allow 3 spaces in a measure for a Front repeat.

IF a rront repeat is followed by a time signature, then

151

there are 2~3/4 spaces between them.

IF a Front repeat is Followed by a key signature. then there are 2-3/4 spaces between them.

IF a Front repeat Follows a bar line. then they have the same location.

Allow 2-1/2 spaces For a time signature.

Ir a time signature is rollowed by a key signature. then there are 2-1/4 spaces between them.

IF a time signature rollows a bar line, t~en there are 1-3/4 spaces between them.

Allow 4 spaces plus one space For each sign but one Tor a key signature.

Allow one space iF the First note in a measure has an acc i denta 1 .

IF the next measure has a cleF change. then allow 5 spaces to print it in the current measure.

Allow 1-1/2 spaces iF the measure has an end repeat.

IF two notes are tied or slurred together and the notes are not on the same line, then create two "slur" nodes 'for the Figure. One node is From the first note to the end of the line and the other is From the beginning of a line to the second note.

IF two notes on the same line are slurred or tied together, then create "slur" node For the slur or tie.

IF a tie is a short tie and the notes are on a line, then the left end point is 3/4 spaces to the right of the notehead of the First note and 1/4 space above or below the notehead (in the direction or the tie).

IF a tie is a short tie and the notes are on a space, then the leFt end point is 1/2 space to the right of the First notehead and 3/4 spaces above/be10w it.

If'a tie is a short tie and the notes are on a line, then the right end point is 3/4 spaces to the lert or the second notehead and 1/4 space above/below it.

152

IF a tie is a short tie and the notes are on a space, then the right end point is 1/2 spaces to the leFt of the second notehead and 3/4 spaces above/below it.

IF a tie is a short tie and the notes are on a line, then the peak is located halFway between the noteheads and 1/2 space above or below them, depending on the direction of the tie.

IF a tie is a short tie and the notes are on a space, then peak of the tie is located halfway between the noteheads and 1 space above or below them.

If a tie is not a short tie and has both notes on a line, then the left endpoint is located 3/4 spaces to the right of the first notehead and 3/4 spaces above or below it (in the same direction as the tie).

If a tie is not a short tie and has both notes on a space, then the left endpoint is located 1/2 spaces to the right of the first notehead and one space above or below it.

If a tie is not a short tie and has both notes on a line, then the right endpoint is located 3/4 spaces to the right of the second notehead and 3/4 spaces above or below it.

If a tie is not a short tie and has both notes on a space, then the right endpoint is located 1/2 space to the right of the second notehead and one space above or below it.

If a then

tie the

noteheads them.

is not a short tie and has both notes on a line, peak of the tie is located halfway between the of the tied notes and 1-1/2 spaces above or below

If a tie is not a short tie and has both notes on a space, then the peak of the tie is located halfway between the noteheads of the tied notes and 2 spaces above or below them depending on the direction of the tie.

A curve is a short tie if it is a tie and the first and last notes are less than 3 spaces apart.

IF a slur or a tie has any connects with a downward stem, concave down.

note between the notes it then the slur or tie is

153

If the left note or a slur is on a line and the stem direction of the note is the same as the direction or the slur, then the lert end point or the s1ur is 1-1/4 spaces directly above or below the notehead (in the opposite direction of the stem).

If the leFt note of a slur is on a space and the stem direction or the note is the same as the direction of the slur, then the left end point of the slur is one space directly above or below the notehead (in the opposite direction or the stem).

If the left note of a slur has an upward stem. and the slur is concave down. then the left end point of the slur is 2 spaces to the right of the notehead and 1/2 space above the top of the stem.

If the right note of a slur is on a line and the stem direction of the note is the same as the direction of the slur, then the right end point of the slur is 1-1/4 spaces directly above or below the notehead (in the opposite direction of the stem).

If the right note of a slur is on a space and the stem direction of the note is the same as the direction of the slur. then the right end point of the slur is one space directly above or below the notehead (in the opposite direction of the stem).

If the right note of a slur has an upward stem. and the slur is concave down. then the right end point of the slur has the same Y coordinate as the n6tehead and is 1/2 space above the top of the stem.

If a slur is concave down, then the peak of the slur is one space above the highest obstacle between the left and right notes or the slur.

If a slur is concave up. then the peak of the peak of the slur is one space below the lowest obstacle between the left and right notes of the slur.

Appendix C

Diagrams or Husfcal Characters

This appendix diagrams or the graphical characters

used by ENGRAVE to produce musical notation.

154

155

Figure C-l Whole Note

10

156

10

Figure C-2 Closed Notehead

'0

157

Figure C-3 Open Notehead

'0

158

10

Figure C-4 Sharp

5

159

Figure C-5 Natural

:s

160

Figure C .. 6 Flat

5

161

8 ia~

/ ~

.........

... =-

- -----~ r"

...,

$ ~ \ I . I

$

162

Figure e·g Repeat

I i~

10

_ S

. I

.5 1.0

I-

-

i-

,.

l

1

....

10

r

r -

-t-

.

- r-

l

163

$ .

$

Figure C*9 Repeat (beginning)

. "20

5

164

Figure C-IO Flag

10

165

I/O

Figure C-l1 Treble Clef

20

166

Figure C .. 12 Bass Clef

'0

161

Figure C-13 Figure '3' for time signatures

168

Figure C-14 Figure '4' for time signatures

169

Figure C-15 Quarter Rest

5

170

Figure C-16 Eighth Rest

Appendix 0

NOTES

Input

expressed

to the

in NOTES

printing system is a piece or music

-- a language ror representing music

on a computer. We describe NOTES brieFly to acquaint the

reader with it and to aid in the understanding or the

system.

NOTES is a list-structured language suitable For use

with PROLOG or LISP. It has the advantage that it is also

reasonably easy For human beings to read.

A score is

elements or the

initializations

represented

1 i st are

such as the

as a list. The rirst

the composer, the title and

key signature and the time

signature. The elements Following are lists For each

movement or section. Each movement is Further decomposed

into parts For each instrument. It is the parts that

contain the important inFormation For printing.

To represent a note, we must First represent both

pitch and duration. A note is a list or at Jeast two

character strings. The First string represents

171

172

duration. Duration is a sequence of digits Followed by

zero or more "d"'s which may be Followed by an optional

"t" and more digits. The First digits Form an integer

that represents the type of note. A halF-note would be

represented by a "2" and a quarter-note by a "4". The

optional "d"'s are used For dotted notes one tid" For

each dot. The "t" shows that the note is part of an

n-tuplet group with the integer Following being n. Pitch

is represented by a three- or Four-character string. The

First character is the letter name of the note. The

second is the octave that the note is in and the third

and Fourth characters. iF present. represent an

accidental. IF the pitch is simply the letter "R". then

the note is a rest. Any other strings Following the

pitch will be modiFiers that represent articulation

markings. Where possible. these are literal indicators of

the Figure.

Measures are simply lists of notes. At the head of

the 1 i st is the word "meas" Fo 1 lowed by any mod i F i ers to

the measure such as a repeat or a key change. See Figure

0-1 For an example of a Few measures expressed in NOTES.

The Following is the NOTES representation of the

173

piece shown in Figure 8.

(part tenor (meas (4 F3)(4 EN3)(4){4 G3» (meas (4)(4 F#3)(8)(8 G#3) (4 A3» {meas (2)(8)(16 G#3)(16 F#3)(4 G3» (meas (4){8 A3)(8 E2)(8 A3)(16 Bb3)(16 A3)

(8 GN3)(8 A3» (meas (8 B3)(8 04)(8 G3){8 A3)(8 B3)(8 G3)

(8 E3)(16 FN3)(16 G3» (meas (4 A3)(2 G3)(4 F3» (meas (8)(16 E3){16 03)(4 A3)(4)(4 G3» {meas (4 A3)(4 R)(2 R» {meas (1 R» (meas (2 R)(2 83» (meas (2 A3){2 C4» (meas (4d BN3)(8 C#4) (2 04» (meas C4d)(16 C#4)(16 BN3)(2 C4» (meas (8 04)(8 E4)(8 04)(8 CN4) (8 BN3) (8 A3)

(8 83)(8 04»)

174

(meas (ad D5) (16 B4) (4 A4) (2 R) )

{meas (4t3 B4) (4t3 C5) (4t3 D5) (2 E5)

(meas (4 R) (4 r#5) (2 E5) )

Figure D-l NOTES Eumple

Appendix E

SpatIal Planner GraphIcs

E. 1 Character Symbo 1 s

Many of the symbols in musical notation are

characters. That is, they have a constant shape and

orientation. These characters were hand digitized and

made available as a sequence of points. A Figure is

speciFied by a list of triples (X Y OP) where X and Yare

spatial coordinates relative to the center of the Figure

and OP is either "move" or "line" indicating whether the

operation

location

should be MOVE_ABS_2 or LINE_ABS_2. When the

of a Figure is calculated, these values are

added to X and Y For each point and the proper operation

is perFormed. Drawings of these Figures are in Appendix

c.

E.2 Beaning

Among the non-character symbols are the beams

between adjacent eighth notes and smaller notes. These

beams vary in length and slope For diFFerent combinations

of notes. The slope of the beam must follow the general

175

176

slope of the notes it connects. We use linear regression

to compute the slope of the notes in a beat-unit and use

this slope as the slope of the beam. The stern directions

of all the notes in the beat-unit are determined by the

-stern-direction that the note Farthest From the middle of

the staff would have if it were not beamed. The rule for

locating the beam'dictates that no stem in the beat-unit

should have a length of less than a sixth (2 1/2 spaces).

We position the beam one octave (up or down

according to stern-direction) from one of the notes in the

beat-unit. This note is chosen so that all of the other

notes in the beat-unit have acceptable stem lengths. For

example, if the notes have up sterns and the beam has a

positive slope. then the note chosen is the highest note

in the beat-unit with ties going to the First note. 5ee

Figure 47 For some examples of beaming.

E.3 TIes and Slurs

Ties and slurs are modeled as conic sections using

the techniques outlined in [Faux and Pratt. 1979, pp.

17-36]. We find that the intersection of two conic

sections, 51 = 0 and 52 = 0 specifies a family of

177

conics given by

(1 - q)Sl + qS2 = 0

In particular,

= 0 be degenerate conics where 1 l' 12 and 13 are

straight lines. The equation

(1 - q)l112 + q13 = 0 (1)

speciFies a Family of conics that are tangent to 11 and

12 at their intersections with 13· See Figure E-l.

For a tie or slur we let the point (xl'Yl) be the

leFt-most point of the curve and -let (x2'Y2) be the

right-most point. We speciFy some peak by (x3'Y3)·

For ties, this peak is the midpoint of the two

noteheads. For slurs, it is slightly higher than the

highest obstacle that must be cleared_ The height is a

Function of the distance of the obstacle From the

midpoint of 13- The closer the obstacle is to the end

of the curve, the smaller the height becomes. To speciFy

the curve uniquely, let ali ne through

and perpendicular (speciFied by

118

~--~

-----,

Figure B-1

The illterseCtioll of two degenerate Collies

179

intersection of' 13 and 14· Now let (Xt,Yt) be

another point of' 14 such that (x3'Y3) is midway

between (xb'Yb) and (Xt ,Yt) • If' 1 1 is a 1 i ne

through (xI'Yl) such that 1 1 and 12 intersect at

(Xt,Yt) then we have speciFied our geometry as shown

in Figure E-2. The equation of 13 is:

13 = a3x + b3Y + c3

= (Y2 - Yl)x + (Xl - x2)Y + x2Yl - xIY2

Since 14 is perpendicular to 13' its slope is

-1 -x2 - xl xl - x2 m4 = = ------- = -------

m3 Yl - Yl Y2 - YI

and its equation is

14 = a4x + b4Y + c4

= (x I - x2)x + (Yl - Y2)Y

+ x3(x2 - X I) + Y3(Y2 - Yl)·

Solving the intersection of 13 and 14 ' we obtain

b3c 4 - b4c 3 xb = -----------

a3b 4 - a4b3

a4c 3 - a3 c 4 Yb = -----------

J33b4 - J34b3

Then

Xt = 2x3 - xb

and

ISO

Pi'Ure B.2

tbe geometry tor c~CU1atillg slurs IIIld lies

181

Yt = 2Y3 - Yb

by our geometry. UsIng these values, we find that

11 = alx + b1Y + c1

= {Yt - Yl)x + (xl - Xt)Y + XtYl - xIYt

and

12 = a2x + b2Y + c2

= (Y2 - Yt)x + (Xt - x2)Y + x2Yt - XtY2

Substituting into (I), we get

(1 - q) 1 11 2 + q 1 32

= Ax2 + Bxy + Cy2 + Ox + Ey + F (2)

aFter some manipulation where

2 A = a 1 a2 ( 1 - a) + a3 q

B = (alb2 + a2b l)(l - q) + 2a3b3q

C = bI b 2(l - q) + b3 2q

0 = (aI c 2 + a2c I)(l - q) + 2a3c3Q

E = (blc2 + b2c I)(1 - q) + 2a3c3q

F = clc2(1 - q) + C32q.

To produce this curve, we take advantage of the fact

that it is relatively Flat. Starting at Xl we

increment X and compute Y unt f 1 we reach x2.

Rearranging (2) to solve for Y we get

Cy2 + (Bx + E)y + (Ax2 + Ox + F) = 0

182

and

-(Bx + E) ± (Bx + E)2 - 4C(Ax2 + Ox + F) Y =

2C

and when C = 0

Ax2 + Ox + F Y =

Bx + E

This geometry creates an ellipse so we use the positive

square root when the curve is concave down and the

negative square root when the curve is concave up.

The geometry we have used works quite well in most

cases. Figure 46 shows an example o~ the ties and slurs

that can be produced.

ENGRAVE - An Expert System that Understands and Generates ... · IntroductIon ENGRAVE is an expert...

Documents

Transcript of ENGRAVE - An Expert System that Understands and Generates ... · IntroductIon ENGRAVE is an expert...