Leon

1

I. ABSTRACT Since the 1950’s, computers have been utilized heavily to further the study of music. Now, music is conceived, recorded, produced, stored, and consumed almost entirely with the assistance of computers. Though music and computers are deeply intertwined in the modern age, there is still a disconnect between music notation and computer representation. As an avid lover of both music and computer science, I became intrigued with this void in computer music. What is the proper way to store and symbolize musical scores? Is there a way to encode scores that do not follow standardized rules of notation? Inspired by both the old and new methods of computer music composition and technology, this project seeks to overcome the boundaries between music notation and digital encoding. Though several established formats for standard digital music encoding exist, it becomes quite difficult to use these formats to encode graphic notation. Scores like György Ligeti’s ‘Artikulation,’ George Crumb’s ‘Spiral Galaxy,’ Baude Cordier’s ‘Belle, Bonne, Sage,’ Brian Eno’s ‘Music for Airports,’ and Karlheinz Stockhausen’s ‘Elektronische Studie II’ all contain unique forms of notation which are nearly impossible to represent using standard means of encoding. How can an unusual shape be encoded in the same way that a standard chord or note would be encoded? Using Stockhausen’s 1954 piece as a proof-of-concept, this paper explores the limitations and capabilities of MEI (Music Encoding Initiative) as a tool for encoding non-standard graphic scores. Using a combination of Python, MEI, and JavaScript, I detail below a method for the encoding and representation of graphic musical notation as well as the creation of an interactive score utilizing this encoding.

II. MEI MEI is an open-source Extensible Markup Language (XML) vocabulary for musical notation. XML, a hierarchical encoding system, can be used to represent virtually any type of document in a simple, machine-readable format. MEI utilizes XML to encode symbolic musical information in a way that can be easily parsed and utilized by computers. For example, the simple musical phrase shown in Figure 1 can be represented as MEI code on the left.

Each piece of information associated with the musical phrase is encoded as MEI data. Though the musical notation and MEI code do not look very similar, they both are equivalently representing the same musical phrase. Similarly, entire scores can be translated and represented using MEI, as demonstrated in Appendix A. Encoding in this format allows more functionality and interaction with the score, as I attempted to execute in this project.

III. MEI’S LIMITATIONS One of the first issues I encountered with MEI was its inability to represent non-standard scores in a suitable format. The glaring issue that stood out to me was the lack of a standard framework for encoding pieces with non-conventional notation, like those found in Appendix B. Since MEI does not currently allow for an easy encoding of this type of

Matthew Leon, Music & Computer Science, University of Virginia Class of 2017

Encoding Non-Standard Forms of Music Notation Using MEI (May 2016)

Figure 1 - A musical phrase represented as MEI markup.

Leon

2

score, I attempted to devise a framework for encoding a non-standard score.

IV. PROJECT OVERVIEW After understanding the limitations of MEI, I wanted to create a project that pushed the boundaries in order to adapt to these limitations. First, after finding a suitable score, I planned to formulate a method for encoding each bit of information in the piece. Using this method, the entire score would be encoded using MEI. Once represented as MEI, a website would be created that could interpret this data and display it as an interactive visual score. Armed with a long-term plan, I set out to find a fitting score.

V. LIGETI’S ‘ARTIKULATION’ The first piece I examined was György Ligeti’s 1958 piece ‘Artikulation.’ This piece is accompanied by Rainer Wehinger’s 1970 graphic notation which was created several years after the composition of the piece, and breaks the piece into several different symbols. The graphic score, shown in Figure 2, is vibrant and interesting to look at, and I attempted to use this piece as my sample encoding. Initially I intended to use scans of the graphic score that would be processed using Adobe Illustrator’s computer graphics to generate Scalable Vector Graphics (SVG) files which could then be linked to MEI. Once encoded, I wanted to create a timeline view of the score which would link each SVG object to a sound from the original piece. After the SVG objects were linked, the score could be interactive and playable, displaying a description of each note event so that information would be available for every note. Each note could also be synthesized in order to hear the note individually. I began the task of analyzing each note of the score, using the visual programming language Max/MSP to process an image of the score and convert the image using spectral analysis, but ultimately this method generated too much noise and was unable to create a clean analysis of the piece. I attempted a few other methods to encode the Ligeti piece, but a few issues arose that made

implementing my plan much more difficult. First, the score was produced after the piece was written, and was not created by Ligeti himself, which means the score is not a perfect representation of the piece. Without a straightforward and exact representation of the piece, an interactive and playable score would be virtually impossible. Furthermore, the piece is laden with ambiguity, since there are many images overlaid atop each other. Using computer graphics image tracing, isolating individual shapes would be very difficult, and I found it to be a fruitless task. Finding this piece too difficult to encode, I decided to pursue another path.

VI. STOCKHAUSEN’S ‘STUDIE II’ After much searching, I settled on Karlheinz Stockhausen’s 1954 piece, ‘Elektronische Studie II,’ pictured in Figure 3. I chose this piece for several reasons. Stockhausen is one of the most influential and important electronic composers of the 20th Century, and several of his works have been critical in fueling my interest in electronic music. In addition, Stockhausen’s piece was one of the first early electronic music compositions to utilize this type of graphic score. The piece was formulaically composed, with relatively distinct and separate segments, allowing me to write a framework for encoding the piece in a much easier way than Ligeti’s piece allowed. Furthermore, Stockhausen’s piece is confusing to understand simply by looking at it; a deep understanding of the piece is necessary in order to

Figure 2 - Rainer Wehinger's graphic score for György Ligeti's 1958 piece 'Artikulation.'

Leon

3

interpret the score, so encoding the piece would allow for easier interpretation.

VII. ELEMENTS OF ‘STUDIE II’ After choosing my piece, I confronted the initial problem with this score: its underlying meaning is obscured by the unique nature of the notation. The score consists of rectangles denoting note mixtures, and triangles denoting dynamics. Though the score looks quite different than standard notation, both hold essentially the same information. Every rectangle is essentially a chord containing five notes, and each triangle details the attack or decay of the chord. Each note mixture has three defining characteristics: pitch, volume, and duration. Pitch is indicated on the vertical axis, shown by the numbers on the left that correspond to frequencies in hertz. The scale of frequencies is based on an interval between each step, which consists of the frequency proportion of the 25th root of 5, as compared to the traditional tempered tuning system with the proportion of the 12th root of 2. Each rectangular note mixture consists of five component frequencies, each equally spaced using this scale. By determining the

lowest and highest frequency of each note mixture, the rest of the component frequencies can be calculated. Initially, I attempted to calculate each frequency from the lowest and highest frequencies listed on the vertical axis, but my results were not consistent with Stockhausen’s results due to Stockhausen’s use of gratuitous rounding. Since oscillators were not very advanced in 1954, Stockhausen often rounded, resulting in varying effects. In order to alleviate this issue, I created an array containing each frequency in the score, and used the index of the highest and lowest frequencies to reference the frequencies stored in the array. The next element, volume, is indicated by the triangles on the bottom of the score. Each shape correlates directly with the envelope of volume for each note, and thus the increase or decrease in volume over time can be calculated by determining the side of the triangle. Finally, the duration of each note is indicated by the numbers along the middle of the score. Each value is shown in centimeters, which corresponds to the length of tape played at 76.2 centimeters per second. Since Stockhausen chose the duration of each note mixture by cutting strips of magnetic tape, each number indicates the length of tape rather than the duration of the mixture in time. In order to calculate the duration of each note

Figure 3 - Karlheinz Stockhausen's graphic score for the formulaically composed 1954 piece 'Elektronische Studie II'.

Leon

4

mixture in seconds, the tape length value can be divided by 76.2.

VIII. ENCODING ‘STUDIE II’ After understanding each independent element of the score, I set about manually encoding the score by identifying five pieces of information for each note mixture: the bottom frequency index, top frequency index, starting position in centimeters, duration in centimeters, and the starting or ending position of the envelope in decibels. Once I had derived each piece of information, I entered them into a spreadsheet (see Appendix C), which was converted into a comma separated value (CSV) file. Each frequency represented in the score was also entered into CSV file, used to populate an array for pitch calculations. Next, the data was parsed by a python script that uses each piece of information from the CSV encoding to calculate useful values which are then rendered as an MEI file. The python code first builds an XML tree with each appropriate element, in preparation for each note mixture’s information to be appended after the CSV file is read. MEI does not contain all elements required to follow my data classification system, so it was necessary to create a few new elements. Beginning with the standard MEI score, the appropriate title, composer, and header information is applied to the document object model (DOM). Next, a new body, mdiv, score, section, measure, staff, and layer are created. Inside the layer, each note mixture is represented as a chord. This chord contains the slope represented as ‘envelope,’ and the starting position as ‘position.’ Using MEI’s ‘tstamp_gs’ attribute, we represent the duration of the chord in seconds. Contained within each chord are the five frequencies in hertz, shown using the ‘freq’ trait. Next, the frequency list is populated by iterating through each frequency in the CSV file. After that, each piece of information from the CSV file of note mixtures is parsed and sent to a function which calculates and creates a note mixture object to be added to an array of notes, called song. The code for CSV parsing code is shown in Appendix D, and the code for note mixture and song class

definitions are shown in Appendix E. Finally, the calculated note mixtures are appended to the XML tree, and exported as an MEI file. The code for XML appending is shown in Appendix F, and a portion of the rendered MEI file is shown in Appendix G.

IX. MAKING THE WEB APP Now that the MEI data was correctly generated, I began the task of doing something useful with the file. I decided to create an interactive version of the score, hosted as a web application. Since this was my first time doing any sort of web development, I first took several online tutorials in HTML, CSS, and JavaScript. Using the JavaScript library jQuery, I first embarked on the task of parsing MEI, using jQuery’s parseXML() function. Once the XML was parsed correctly, I included the image of the score as the background of the web page, and overlaid invisible buttons on top of each note mixture. I created a class called hover that changed the transparency and color of the buttons when hovering on top of them, resulting in an interactive experience with the score. I then linked each button to the corresponding note mixture, using jQuery’s dialog function to display the information parsed from MEI. This implementation, while it looked adequate on the surface, had several issues. One major issue with this implementation was the hard-coded nature of each button. Since I hard-coded each pixel value on the image, the buttons appeared in different locations depending on the screen size, and would move outside of the proper locations when zoomed out. In addition, the buttons were difficult to overlay on top of the envelope triangles, so I decided to pursue another method. Using an HTML image map generator and James Treworgy’s ImageMapster plugin, I was able to link every object on the score and create a more interactive experience. Using this plugin, the website allows the selection of both note mixtures and envelopes, as well as a link between each note mixture and its corresponding envelope. More than one dialog can be open at one time, allowing note mixtures to be compared and to be understood more deeply. Once the image mapping was implemented, I added audio functionality by embedding the

Leon

5

original audio file into the website and using HTML’s audio player so that Stockhausen’s 1954 realization can be played and understood in the context of the score. I then added more interactive elements in the dialog box, allowing audio playback and synthesis. Using the duration and starting position elements from the MEI file, each note mixture has a starting point and duration that is linked to the recording. When the ‘play’ button is clicked, the note mixture is played from the context of the recording, beginning at the starting position and ending after the duration of the note mixture has passed. Using HTML’s AudioContext, I added synthesis capabilities, so that each note mixture can be synthesized independently from the recording. Each frequency parsed from MEI can be synthesized one-by-one, or all frequencies can be synthesized together to give an impression of the whole note mixture. An example of a highlighted note mixture and its accompanying dialog is shown in Figure 4. The web application can be accessed at the following link: www4.iath.virginia.edu/mei/ML

X. ISSUES There are a few issues I encountered in the playback of the score that I was unable to correct. Due to inconsistency between Stockhausen’s recording and the score, some note mixtures are slightly skewed. This becomes evident during playback of some note mixtures that contain pieces of other note mixtures that should not be overlapping. Even with scrupulous editing of the sound file, overlapping still occurred. I have not yet resolved a way to fix this problem, other than cutting the recording into smaller sound files to alleviate the issue.

XI. CONCLUSION Currently, only one page of the twenty-six-page score is encoded and interactive. With a bit of time it would be relatively easy to encode the full piece, since a framework for encoding has already been established. In the future, I would like to add more interactive synthesis elements, such as selectable pitches so that the score can be heard with and without certain frequencies. I would also like to add play-through functionality so that the entire piece can be automated and each note mixture will highlight as the piece plays. This project serves as a proof-of-concept of the capabilities of MEI. Though this piece was more straightforward to encode than some other graphic scores, by breaking any score into its fundamental pieces and creating a program to encode each piece, any score has the potential to be encoded using MEI. In the future, I would like to attempt to encode a different score, possibly one with more difficult elements and characteristics.

Figure 4 - A screenshot of the web app, demonstrating a highlighted note mixture and its accompanying dialog. The dialog displays each component frequency along with envelope, position and duration. The ‘Play’ button plays the note mixture in the context of the original recording, beginning at the position parameter and playing for the duration listed in the dialog. The ‘Synthesize All’ button plays a synthesized version of all the component frequencies together, while the following ‘Synthesize’ buttons play the synthesized component frequencies individually.

Leon

6

Appendix A

Figure A.2 - 'Spiral Galaxy' from Volume I of George Crumb's 1972 composition 'Makrokosmos.' This piece's spiral orientation is quite interesting, with features conventional notation simply twisted in a spiral pattern. Since the musical semantics intended by the spiral pattern are not clear, I moved on to examine other potential pieces.

Figure A.3 - ‘Belle, Bonne, Sage,’ composed by Baude Cordier features different colorations and a heart-shaped score. The red notes indicate variations in note lengths from their normal lengths. The piece, focusing on the theme of love, has heart-shaped note clusters, and would also be interesting to encode. Yet the piece, like Crumb’s, uses conventional notation in an unconventional manner that is understood to be beyond the scope of MEI encoded.

Figure A.1 - An example score in standard musical notation (on the left) and a “full” encoding of that score in MEI markup (on the right). The MEI encoding is “full” in the sense that, if done properly, it captures the components of the score that can be considered to precisely represent the intended music.

Leon

7

Appendix B

Figure A.4 - A diagram of Brian Eno's 1978 ambient album 'Music for Airports.' Despite my fondness for the recordings and the intriguing accompanied diagram, it is difficult to interpret as a score since each diagram represents a system used to make music, rather than individual notes. Using this system, it would be very difficult to encode based on the massive amount of controlled randomness.

Figure B.1 - A spreadsheet including the encoded data from the first page of Studie II. The first column indicates the bottom index of the note mixture, which corresponds to the index in an array containing the note mixture’s bottom frequency. The second column indicates the starting position of the note mixture in centimeters. The third column indicates the top index of the note mixture, which similarly corresponds to the index in an array containing the note mixture’s top frequency. The fourth column indicates the duration of the note mixture in centimeters. The fifth column indicates the dB of the ending envelope slope.

Figure B.2 - The python code to parse through both data files. First, the frequency list file is parsed, populating an array with each frequency indicated in Stockhausen’s instructions. Next, a song() object is created to hold each note mixture. For each line in the encoded-data spreadsheet (see Figure B.1), a new note mixture is created, with each relevant piece of information calculated by the calc_frequencies() method. Finally, the new note mixture is added to the song object.

Leon

8

Figure B.3 - The python code’s class definitions for note_mixture and song objects. Each note_mixture object has five frequencies, duration, envelope slope, and starting position. These parameters are calculated and set in the calc_frequencies() method, which takes the raw data from the ‘markup.csv’ file and performs the appropriate calcuations and value assignments for each piece of information.

Figure B.4 -The python code for creating the XML tree. For each note_mixture in the song, a chord is created which contains duration as tstamp_gs, envelope, position, and each frequency as a note. The tree is then exported as ‘meifile.mei.’

Leon

9

Figure B.5 - The rendered MEI file, containing the pertinent information for each note mixture and for the piece of Studie II shown in Figure 1.

Leon

10

WORKS CITED

Ligeti, György, and Ranier Wehinger. Artikulation: Elektronische Musik = Electronic Music. Mainz:

Schtt, 1994. Musical Score.

Madden, Blake. "Brian Eno – “Ambient 1: Music For Airports”." Trust Me I'm a Scientist. N.p., 3 Mar.

2014. Web. 07 June 2016. <http://www.trustmeimascientist.com/2014/03/03/brian-eno-ambient-1-

music-for-airports/>.

Muchmore, Pat. "Scoring Outside the Lines." Opinionator Scoring Outside the Lines Comments. N.p., 3

Aug. 2011. Web. 12 May 2016. <http://opinionator.blogs.nytimes.com/2011/08/03/scoring-outside-the-

lines/>.

"Music Encoding Initiative." MEI. Music Encoding Initiative, 2016. Web. 12 May 2016. <http://music-

encoding.org/>.

Stamp, Jimmy. "5 1/2 Examples of Experimental Music Notation." Smithsonian. N.p., 5 June 2013. Web.

12 May 2016. <http://www.smithsonianmag.com/ar ts-culture/5-12-examples-of-experimental-music-

notation-92223646/?no-ist>.

Stockhausen, Karlheinz, Nr. 3 Elektronische Studien. Studie II. Wien: Universal Edition, 1956. Print.