François Pachet, Active Listening : What is in the Air

8/14/2019 Franois Pachet, Active Listening : What is in the Air

1/16

FranoisPachet,ActiveListening:Whatis in theAir?SonyCSLinternalReport,

1999.

1

MusicListening:WhatisintheAir?

FranoisPachetSONYComputerScienceLaboratoryParis

6,rueAmyot

75005Paris-France

[email protected]

Abstract

TheXXth century is full of technological inventions that made the very idea of a

listeningdevicepossible,fromtheearlygramophonestothelatestportableminidisk

players.Whatevolutions canwepredictforthe listening devicesofthe future, and

how these evolutions will change theway we access and listen to music ? In this

chapter,wesuggestthatlisteningdevicescanbegreatlyenhancedbyprovidingnew

formsofusercontrolswhich provideuserswith semanticallypreservingvariations.

Thesecontrolsareintendedtoallowlistenersdifferentmusicalperceptionsonapiece

ofmusic,byoppositiontotraditionallistening,inwhichthemusicalmediaisplayed

passively by some neutral device. The objective is both to increase the musical

comfortoflisteners,and,whenpossible,toprovidelistenerswithsmootherpathstonewmusic(musictheydonotknow,ordonotlike).Thischapterillustratesthisidea

onafewexamplesofactivelisteningprojectsconductedatSonyComputerScience

Laboratory, Paris, based on the notion of constrained exploratory space. These

constrainedspacessuggestthattheclassicalboundariesbetweencomposing,listening

and mixing may be redefined, thereby assigning new roles to composers, sound

engineersandlisteners.

1. FromButtonstoExploration

Weproposetheideaofexploratorylisteningenvironments,asanaturalevolutionin

thehistoryofmusicalcontrols.Wefirstsketchabriefhistoryofmusicalcontrols,andthenintroducethenotionofsemantic-preservingmusicalexploratoryenvironment.

1.1 HistoryofMusicalControls

Eachtechnologicaladvancehasbroughtwithitnewformsofcontrols.Theoriginsof

listeningmachineswithmass-producedmusicalmaterialsmaybetracedbacktothe

Phonograph,inventedbyThomasEdisonin1878,whichusedtinfoilcylinders,and

shortlyaftertheGramophone,inventedbyBerlinerin1888,whichusedflatdisks.In

thesedevices, therewas nocontrol intentionallygiven tothe user (se,e.g.Read&

Welch, 1976). Therewas, however,an unintentionalcontrol intheGramophone in

thatthehorncouldbeturnedaround,therebyinfluencingthedirectivityofthesoundsource.Electricity soonbeganto beusedfor listening devices,bothwith radio and


2/16


1999.

2

with new electrically recorded disk players in the 20s. The use of electricity also

introduced newcontrols: the volume button and the treble/bass button. Juke-boxes

were introduced in1927, allowing listeners to select explicitlymusic titles fromagiven catalogue of disks, using various sorts of push buttons. The next big

technological advance was the invention of binaural (stereo) recording method in

1931. Thecorrespondingcontrolwasthepanoramicbuttonallowingtocontrolthe

amount of signal inone loudspeakeror the other. Finally, digital format for audio

introduced more controls, e.g. on the equalization of sound. In all these cases,

technologicaladvanceswerefollowedbytheintroductionoftechnicalcontrols,i.e.

controlsoperatingdirectlyonthetechnology(seeFigure1).

Figure 1. A Phonograph (Edison, 1978, left); a Gramophone (Berliner, 1988,

middle),aRock-Ola120-selectionJuke-Box,andaMinidiskplayer(Sony,1997,

right). Advances in technology do not necessary imply more intelligent user

control.

1.2 AMatterofSemantics

Theverynotionofmusicalcontrolraisestheissueof semantics.Theissueofmusical

semantics - doesmusichavemeaning? - has been longdebatedbymusicologists,

leading todifferent theories,whichusuallyparalleled the theoriesof semantics for

languages.Oneofthemaindistinctionmadebytheoristsistheoppositionbetween

so-called referentialists and absolutists. Referentialists claim that musical

meaningcomesfromactualreferencesofmusicalformstooutsideobjects,i.e.music

meanssomethingwhichisexternaltomusicitself.Forinstance,aparticularscalein

Indianmusicmay have a reference to a particular human mood. Absolutists, e.g.

Strawinsky, claimon the contrary that themeaningofmusic, ifany, lies inmusic

itself,i.e.intherelationsentertainedbymusicalformstogether.Althoughthesetwoviewpointsarenotnecessarilyexclusive,asnotedbyMeyer(Meyer,1956),theyleave

openmuch of the questionofmeaning. EugeneNarmour elaborated amuchmore

precisetheoryofmusicalmeaningbasedonthepsychologicalnotionofexpectation

(Narmour,1992).Inthistheory,meaningoccursonlywhenmusicalexpectationare

deceived.On the otherhand,Rosenargues (Rosen, 1994)that the responsibilityof

preservingthemeaningofamusicalpieceliesonlyintheperformeritself,whohasto

choosecarefullyamongainfinitesetofpossibleinterpretationswhichoneisclosestto

theoneintendedbythecomposer.

Without committing toone particular theory ofmusicalmeaning,wecan note that

meaning-whateveritmeans-hastodowithchoosingamongasetofinterpretations

the right oneor the right ones, i.e. those intendedby the composer.Asecondremarkisthatthecontrolsgivenbythehistoryofsoundrecordingtechnologyhave


3/16


1999.

3

neverhadanyconcernaboutmusicalsemantics:whatdoesitmeantoraisethesound

levelofa record ? toshiftthe signal tothe left loudspeaker? toincrease thebass

frequency ? Are the intentions of the composers, or even of the sound engineers,preservedinanyway?

From this remark, we suggest that interesting musical controls should preserve

some sort of semantics of the musical material, i.e. preserve intentions,whenever

possible.Weargue thatmoremeaningful controls, in thecontextofmoderndigital

multimedia technology,amount toshiftingfromtraditionalbutton-based technology

tomusicalexplorationspaces.

1.3 MusicInteractivity

Aswehaveseen,technologicalbuttonsbearnosemantics,becausetheyaredirectlygroundedonthetechnology,withoutanymodelofthemusicbeingplayed.Butwhat

canbesuchamodel?

Interestingapproachesinmusicalinteractivityarethemusicnotationsystems,inthe

contextofannotationofmusicdocuments,asintheworksofLepain(1998),orinthe

Acousmographsystem(INA-GRM).Inthesesystems,theprimaryissueaddressedis

notmusiclisteningperse,butrathermusic notation,i.e.howtorepresentgraphically

amusicaldocument(thedocumentitselfortheperceptionofthedocument),orhow

toinferamodelofthemusicwhichcanbenotedorrepresentedgraphically.

Another answer may be found in the notion of open form, initially developed in

literature(Eco,1962),whichhashadmuchimpactonmusictheoryandcomposition

(Stockhausen,Boulez).Theideaofmusicalopenformisthatthecomposerdoesnot

createaready-to-usescore,butratherasetofpotentialperformances,whichcanbe

seenasamodel ofscores,as explainedby (Eckel,1997):Music isnotany longer

conceived in form of finite units but in terms of models capable of producing a

potentiallyinfinitenumberofvariantsofaparticularfamilyofmusicalideas.The

selectionorinstantiationoftheactualscoretobeplayedisdelegatedtotheperformer.

In recent incarnations of open form, it is the listener himself who instantiates the

model, as for instancein theCave (Cruz-Neiraetal.,1993)orCyberStage (Eckel,

1997).Inthesecases,theuserisimmersedinarealisticvirtualenvironment,andhas

the control on his position andmovement in a virtual world. Hismovements are

translatedintovariationsinthemusicalmaterialbeingheard.Theseapproachesmay

beconsideredasradical,inthesensethattheuserhasagreatdealofresponsibilityinmakingthemusic.However,theissueofsemanticsisnotdirectlyaddressed,sincethe

modelinprincipleisunder-designed,i.e.allpossibleexplorationsarealwayslicit,

whatevertheymaybe.Inthisrespect,thereisastrongrelationbetweenopenform

virtual environments and programming languages for music composition, such as

OpenMusic(Assayagetal.,1997),CommonMusic(Taube,1991)orElody(Orlareyet

al.1997).Intheseapproachesindeed,thegoalistoproposetheusertoexplorespaces

withasmuchfreedomaspossible,andnotconstraintheuserinspecificareas.

1.4 ActiveListening

ActiveListeningreferstotheideathatlistenerscanbegivensomedegreeofcontrolonthemusictheylistento,thatgivesthepossibilityofproposingdifferentmusical


4/16


1999.

4

perceptionsonapieceofmusic,byoppositiontotraditionallistening,inwhichthe

musicalmediais playedpassively by someneutraldevice.The objective isbothto

increasethemusicalcomfortoflisteners,and,whenpossible,toprovidelistenerswithsmoother paths to new music (music they do not know, or do not like). Active

listeningisthusrelatedtothenotionofopenformoutlinedabovebutdiffersbytwo

important aspects: 1)we seek to create listening environments for existingmusic

repertoires, rather than creating environments for composition or free musical

explorationand 2)weaimatcreating environments inwhich thevariationsalways

preserve the original semantics of the music, at least when this semantics can be

defined precisely. For us, the issue if therefore not to introduce yet another

technological button in the interface of the listening device, but rather to design

buttons that make sense, thereby breaking the long tradition of technological

buttonsinitiatedbyEdison.

What sense,what meaning arewe talking about?How canmusiccontrolsbe

designed to trigger semanticpreserving actions ? The answer stems from the new

landscape of music recording created by digital multimedia, sketched in the next

section.Wewillthenillustrateourideasbytwoexamplesofactivelisteningprojects

atSonyComputerScienceLaboratory-Paris.

2. TheNewFactsofMultimedia

Digitalizationofmultimediadatahasanumberoftechnicaladvantageswhicharewell

knowntoday:bettersoundquality,bettercompression,losslesscopy,etc.Theaimof

thischapteristoshowthatdigitalizationofmultimediadataalsoinduce-evenina

stillpotentialform-anumberofrevolutionsinthewaymusicmaybeaccessedand

listenedtobyendusers.Wewilloutlinethreeoftheserevolutions,whichformthe

basisofourargumentation, focusingon theparadigmatic shifts they convey,rather

thanontechnicalaspects.

2.1 StructuredAudio:HomeasaReconstructionMachine

Theideaofstructuredaudiohasinitiallybeendevisedtoallowbettercompressionof

highqualityaudio.StandardizationeffortsliketheMpeg-4projectembodythisidea,

andtrytomakeitpracticalonalargescale(see,e.g.theMachinelisteningGroupof

theMedialab,Sheireretal.,1998).

Theideaissimple:insteadoftransmittingaready-to-listensound,onlyadescription

ofhowtomakethesoundistransmitted.Theactualsoundisreconstructedathome,

oratthelistenerslocation,providedofcoursehe/shehastherightsoftwaretoprocess

this reconstruction properly. Structured audio actually extends this basic idea to

include fully-fledged scenedescriptions, that is, not onlydescriptions ofindividual

sounds,but descriptionofgroupsofsoundsplayingtogethertomakeupa pieceof

music.Theactualtechnicaldetailsofscenedescriptionalsoincludeallwhatisneeded

toreconstructasoundorpieceofmusicrightfully,e.g.effects,adaptationtothelocal

soundreproductionsystem,andsoforth.

Inourcontext,wearguethatthenotionofscenedescriptionopensupnewdoorsfor

meaningful controls. Indeed, since the music is delivered as a kit, lots ofpossibilitiescanbeimaginedtoinfluencethewaythekitisactuallybuilt,accordingto


5/16


1999.

5

userpreferences.Ofcourse,thesevariationsaroundhowthekitshouldbeassembled

havetobecoherent,whicharepreciselythematterofourwork.

2.2 Meta-dataandAllThatJazz

Thefactthatmusicaldataisnowproduced,codedandtransmittedinadigitalform

hasnumerousandwell-knownadvantages:bettersoundquality,possibilityoflossless

transmission and copying (thereby raising new copyright problems). An important

nontechnicalconsequenceisthepossibilitytoencodenotonlythemusicitself-the

digitalized sound - but also any sort of symbolic information. Such symbolic

informationmay be used to code and transmit data on themusic itself, so-called

informationoncontent,meta-dataoralsobitsaboutbits.

Whywouldonewant totransmit suchmeta-data ?The interestsareobvious inthecontext of document indexing. If musical data is accompaniedwith corresponding

adequate descriptions, digital catalogues can then be accessed using sophisticated

query systems. Current standardization efforts like Mpeg-7 embody this idea

(MPEG7,1998),andtrytodefinestandardsfordescribingmeta-dataforallsortsof

multimedia documents. MPEG-7 aims for instance at making the web more

searchable for multimedia content than it is today, make large content archives

accessibletothepublic.

Here again, we would like to emphasize the conceptual rather than the technical

aspectsofthisparadigmshift:meta-dataopensalsodoorsforimaginingnewlistening

systemsinwhichtheusermayaccessdatainadrasticallydifferentway.Insteadof

being a passive, neutral support, music becomes an active, self-documentedknowledgebase.Again,whatkindoflisteningdevicescanbeimaginedthatexploit

thisinformation?

2.3 SizeofDigitalCatalogues

Digitalization ofmultimedia data has yet another consequence: the availability of

hugecataloguesofmultimediadatatousers.Inthecaseofmusic,thereis,herealso,a

conceptualshiftwhichhasnothingtodowiththetechnologyoflargedatabases.The

mainissueraisedbythistechnologicaladvanceishowtoaccesshugecataloguesof

music,notfromatechnicalviewpoint,butfromausersviewpoint.Recallthejuke

box,inventedinthelate20s:atypicaljukeboxwouldcontainabout120titles,whichis the size of an average users discotheque. Browsing through all the titles was

probably part of the pleasure, and selection could be made just like at home: by

choosingoneitemoutofacollectionofitems,whichatleasttheuserhasseenonce.

Now a typicalcatalogueofamajor companyis about50.000 items.Whathappens

whenthecollectiontoselectfromissuchacatalogue?Evenmoreterrifying,what

happensifalltherecordedtitlesbecomeavailablethroughnetworkstousersathome

? Estimating the total number of all recorded music is difficult, but it can be

approximated to about 2 million titles (see, e.g. the size of MusicBoulevard or

Amazon databases). The figure can be probably doubled to include nonWestern

music. Every month, about 4000 newCDs are issued on themarket. It is clearly

impossible toapply usual techniques ofmusic selection in this new context.Whatdoesitmeantolookforatitlewhenthemassoftitlesissohuge?


6/16


1999.

6

3. Spatialization:TheMusicSpaceProject

Thefirstparameterwhichcomestomindwhenthinkingaboutusercontrolonmusicis the spatialization of sound sources.We conduct a project for investigating the

technical and conceptual issues related to meaningful user-control of music

spatialization,calledMusicSpace.

3.1 MotivationandDescriptionofMusicSpace

InMusicSpace,theusercanlistentopiecesofmusicusinganinterfaceinwhicheach

instrument inthe pieceis represented by agraphicalobject (seeFigure 2).Moving

these objects around modifies the mixing of sound sources in the global sound.

Moreover,anobjectrepresentingthelistenerhimself-avatar- isalsorepresentedin

theinterface,sothatallthemixingparameters(volume,panoramicposition,etc.)arecomputedaccordingtotheavatarsposition.Thebasicsystemprovidesthepossibility

of 1) moving around the avatar, to induce amixing as if the listenerwasmoving

around theactualmusical setup,and2)movingaroundtheinstruments themselves,

therebyinducingadifferentmixingasifthelistenerwasasortofsoundproducer.

Experimentationsofthisbasicsystemwereconductedonaveragelistenersandmusic

composers.Itclearlyappearedthatalthoughthephysicalactionsofmovingavataror

instrument icons around in a window are very similar, the possibility of moving

aroundlistenersavatarsisquitedifferentconceptuallythanthepossibilityofmoving

around instruments. Indeed,moving theavatarcorresponds totheactionofmoving

oneselfaroundamusicalsetting.Movinginstrumentscorrespondtoamoretechnical

viewonthemusic-thesoundengineersview.Thissecondpossibilityappearedtosomeusersasheretic,sinceitpracticallygivesusersthepossibilityoftotallychanging

theoverallmixingofthemusicalpiece!

The second phase of our project consisted in introducing a way of somehow

constraining user actions, to avoid situationswhere the mixing produced is totally

unrelatedtotheoriginalspiritofthemusic(Pachet&Delerue,1998).Weproceeded

byintroducinga particulartechnique,calledconstraintperturbation,whichprecisely

allows instruments tobe linked togetherby relations that are always enforced: the

systemusestheseconstraintstopropagatechanges,sothatthesetupalwaysremain

consistent.Forinstance,arelatedconstraintmaybesetbetweenthedrumandthe

bass, so that one ofthem ismovedcloser to the listeners avatar, the otherone is

moved accordingly (with the same distance ratio). On the contrary, a balanceconstraintmaybesetbetweentwosoundsourcesthatshouldalwaysbemutuallyin

opposition: for instance, when the chorusing instrument is brought closer, the

accompanimentismovedaway.Theseconstraintscanfinallybecomposedtogetherto

createrichenvironmentsinwhichusersmaychangetheinstrumentpositions,butthe

constraint systemensures thatthe overallmixing alwaysremainconsistentwith the

engineerorcomposerconstraints.


7/16


1999.

7

Figure2.TheinterfaceofMusicSpace. Instrumentsarerelatedbyconstraints.

The avatar as well as instruments can be moved around by the user. The

constraintsembodyanautomaticsoundengineer.

3.2 ExplorationSpace

There are twoways to interpretMusicSpace. One is to see it as an embodiment -

simplisticbutoperational-ofasoundengineer:theusermaymovesoundsusinghigh

level, simple actions; the system corrects these actions by moving other sound

sources according tohis knowledge ofsoundmixing.This knowledge isexplicitlyrepresentedasconstraints.

The otherviewpoint is toseemixing constraintsasanontologyofmixing actions,

whichallowstomixintermsofpropertiesofsetups,ratherthanintermsofatomic

actions on knobs and faders. This ontology allows to specify properties of

configurations, which are guarantied to be always enforced, rather than specify

explicit configurations. In this respect, constraints represent a semantics of sound

source configuration, and the resulting - constrained - exploration space allows to

explorevariousconfigurationswithoutviolatingthespiritoftheoriginalmixing.

MusicSpaceisalsotobeseenasanexampleofexploitationofreconstructedmusic.

Asoutlined inSection2.1, future standardswilldelivermusicby chunks, possibly

transmitting sound sources separately, together with specifications on how to

reconstructthemusicwholefromtheparts.Constraintsareonewayofspecifyingthis

reconstruction, which nevertheless leaves room for new semantic-preserving user

control.Assuch,itisaradicallynewformofGramophone,asdescribedin1.1:not

only does MusicSpace provide more refined controls on sound spatialization than

turningthehornaround,butthesecontrolspreservetheunderlyingintentionofsound

sourceconfigurations.

4. MusicCatalogueAccess

Theissueofmusicdeliveryconcernsthetransportationofmusicinadigitalformatto

users.Musicdeliveryhasrecentlybenefitedfromtechnologicalprogressinnetwork

transmission,compressionofaudio,andprotectionofdigitaldata(Memon&Wong,


8/16


1999.

8

1998).Theseadvancesallownoworinthenearfuturetodeliverquicklyandsafely

musicto usersinadigital format throughnetworks,either internetor digital audio

broadcasting.

Moreover, as seen inSection 2.2, digitalizationof datamakes itpossible today to

transportinformationoncontent,andnotonlydataitself.Together,thesetechniques

give the users, at home, access to huge catalogues of annotatedmultimedia data,

musicinparticular.Thesetechniquesaimatsolvingthedistributionproblem,i.e.how

to transport data quickly and safely to users. Paradoxically, these technological

advances also raise a newproblem for the user: how to choose among suchhuge

catalogues?

4.1 MotivationandIdeas

Fromtheuserviewpoint,accessingalargequantityofmusicindeedisproblematic:it

cannotbereducedtoa simpledatabaseproblem,because,bydefinition,usersdonot

know precisely what they look for. The problem of choosing items is general in

westernsocieties,inwhichthereisaneverincreasingnumberofproductsavailable.

Forentertainmentandspeciallymusicthechoosingproblemisspecific,becausethe

underlying goals - personal enjoyment and excitement - do not fall in the usual

categories of rational decision making. Although understanding a users goals in

listeningtomusicisverycomplexinfullgenerality,wecansummarizetheproblemto

twobasicandcontradictoryingredients:desireofrepetition,anddesireofsurprise.

Thedesireof repetitioniswellknowninmusictheoryandcognition.Experimental

psychologyshowstheimportanceofrepetitionsinmusic.Atthemelodicorrhythmiclevelsofmusicrepetitionbreedscontent.Forinstance,sequencesofrepeatingnotes

create expectations of the same note to occur. At a higher level, tonalmusic, for

instance, is basedon structures that create strong expectations or the nextmusical

events tocome (for instance, adominantseventhchord createsanexpectation ofa

resolution). Music theorists have tried to capture this phenomenon by proposing

various theories ofmusical perception based on expectationmechanisms (see e.g.

Meyer,1956),particularlyformodelingtheperceptionofmelodies(Narmour,1992).

At themore global levelofmusicselection, this desire of repetition tends tohave

people wanting to listenmusic that they know already (and like) or music that is

similartomusictheyalreadyknow.Forinstance,aBeatlesfanwillmostprobablybe

interested in listening the latest Beatles bootleg containing hitherto unreleasedversionsofhisfavoritehits.

Ontheotherhand,thedesireforsurpriseisakeytounderstandingmusic,atalllevels

ofperception.Theverytheoriesthatemphasizetheroleofexpectationinmusicalso

show that listeners do not favor expectations that are always fulfilled, and enjoy

surprisesanduntypicalmusicalprogressions(seee.g.SmithandMelara,1990).Ata

largerlevel,listenerswantfromtimetotimetodiscovernewmusic,newtitles,new

bands, or new musical genres. This desire is not necessarily made explicit, but is

neverthelessasimportantasthedesireforrepetition.

Of course, these two desires are contradictory, and the issue inmusic selection is

preciselytofindtherightcompromisebetweenthesetwoforces:provideuserswith

items they alreadyknow, and provide themwith items theydonot know,butwill

probablylike.


9/16


1999.

9

Fromtheviewpointofrecordcompanies,onegoalofmusicdeliveryistoachievea

betterexploitationofthecatalogue.Indeed,recordcompanieshaveproblemswiththe

exploitation of their catalogue using standard distribution schemes. For technicalreasons,onlyasmallpartofthecatalogueisactuallyactive,i.e.proposedtousers,

intheformofeasilyavailableproducts.Moreimportantly,theanalysisofmusicsales

shows clearly decreases in the sales of albums, and short-term policies based on

selling lots of copies of a limited number of items (hits) seem to be no longer

profitable.Additionally,thesalesofgeneral-purposesamplers(e.g.BestofLove

Songs)are nolonger profitable,either becauseusershavealready the hits intheir

owndiscotheque,orbecausetheydonotwanttobuysamplersinwhichtheylikeonly

afractionofthetitles.Exploitingmorefullythecatalogueshasbecomeanecessityfor

recordcompanies.Insteadofproposingasmallnumberofhitstoalargeaudience,a

natural solution is to increase diversity, by proposing more customized albums to

users.

4.2 ApproachesinMusicSelection

Current approaches inmusic selection can be split up in two categories: 1) query

systems for accessing music catalogues, and 2) recommendation systems for

proposingnoveltitlestousers.Inbothcases,theseapproachesprovidesetsofitemsto

theuser,whichhe/shehasstilltochoosefrom.

Querysystemsaddressmainlydatabase issues forstoringandrepresentingmusical

data.They proposemeans of querying musical items using some sort of semantic

information.Variouskindsofqueriescanbeissuedbyusers,eitherveryspecific(e.g.

the title of the Beatles songwhich contains theword pepper), or largely under

specified(e.g.Jazztitles).

Collaborative filtering approaches (Shardanand, and Maes, 1995) aim primarily at

achievingthesurprisegoal,i.e.issuerecommendationsofnoveltitlestousers,with

thehopethattheserecommendationswillbeenjoyed.Collaborativefilteringisbased

ontheideathattherearepatternsintastes-tastesarenotdistributeduniformly.This

ideacanbeimplementedverysimplybymanagingaso-calledprofileforeachuser

connected to the service. The profile is typically a set of associations of items to

grades.Forinstance,intheMyLaunchsystem,gradesvaryfrom0(Ihateit)to5(this

is my preferred item). In the recommendation phase, the system looks for all the

agentshavingasimilarprofiletheusers.Thissimilaritycanbecomputedeasilybyadistancemeasureonprofiles,such asahammingdistance. Finally, the systemwill

lookforitemslikedbythesesimilaragents,whicharenotknownbytheuser,and

recommendstheseitemstohim/her.Typicalcollaborativefilteringsystemsformusic

aretheFireflysystem(Firefly,1998),MyLaunch(MyLaunch,1998),theAmazonweb

site(Amazon,1998),orthesimilarityengine(Infoglide,1998).

However,therearelimitationstothisapproach.Theselimitationsappearbystudying

quantitative simulations of collaborative filtering systems, using simulations

techniquesinspiredfromworksonthedisseminationofculturaltastes(Epstein,1996;

Cavalli-SforzaandFeldman,1981).

Thefirstoneistheinclinationtoclusterformation,whichisinducedbythevery

dynamics of the system. The experimental results achieved so far show that such

systemsproduceinterestingrecommendationsfornaveprofiles,butgetstuckassoon


10/16


1999.

10

as the profiles get bigger (about 120 items): eclectic profiles are somehow

disadvantaged.

Anotherproblem, shown experimentally, is that thedynamics inherently favors the

creation ofhits, i.e. itemswhichare likedbyahugefractionof thepopulation.Of

course,theexistenceofhitsisnotabadthinginitself,buthitsneverthelesslimitthe

probabilityofotheritemstosurviveinaworlddominatedbyweightsums.

In short, collaborative filtering is ameans of buildingsimilarity relations between

items, basedonstatistical properties of groups ofagents.Assuch, it addresses the

goalofsurprise,inasafeway,byproposingusersitemswhicharesimilartoalready

knownones.However,clusterformationandunevendistributionofchancestoitems

(e.g.formationofhits)arethemaindrawbacksoftheapproach,bothfromtheusers

viewpoint (clusters fromwhich it isdifficult toescape),and the content providers

viewpoint(nosystematicexploitationofthecatalogue).

4.3 On-the-flyMusicProgramGeneration

TheRecitalComposer Project (Pachet etal., 1999)is basedon a radicallydifferent

approachtomusicselection: insteadofproposing users setsofindividual titles,we

proposetobuildfully-fledgedmusicprograms,i.e.sequencesofmusictitles.

There areseveralmotivations forproducingmusicprograms, rather than unordered

collectionsoftitles.Oneissimplybasedontherecognitionthatmusictitlesarerarely

listenedtoinisolation:CDs, radioprograms,concertsare allmadeupof temporal

sequencesofpieces,inacertainorder.Thisorderismostofthetimesignificant,i.e.

differentordersdonotproducethesameimpressionsonlisteners.Inaway,thewhole

craftofmusicprogramselectionispreciselytobuildcoherentsequences,ratherthan

simplyselectindividualtitles.

Thesecondmotivation is thatpropertiesofsequencesplayanimportantrole inthe

perception of music: for instance, several music titles in a similar style convey a

particular atmosphere, and create expectations for the next coming titles. As a

consequence, an individual title may not be particularly enjoyed by a listener in

abstracto,butmaybetherightpieceattherighttimewithinasequence.

Rather than focusingon similarityofindividual titles,wecan exploitpropertiesof

sequencestosatisfythethreegoalsofmusicselection.Theproposalisthereforethe

following.Firstwebuildadatabaseoftitles,withcontentinformationforeachtitle.Thenwespecifymusicprograms by giving the properties orpatternswewant the

program to have. These properties are represented as constraints, in the sense of

constraintsatisfactiontechniques.Finally,aconstraintsolvercomputesthesolutions

ofthecorrespondingcombinatorialpatterngenerationproblem.

Theproblem,aswedefineit,isthereforetobuildmusicprograms,seenastemporal

sequences of titles, inorder to satisfy the threegoals ofmusic selectionproblem:

repetition,surprise,andfullexploitationofthecatalogue.Asanexample,wewilltake

amusicprogramforwhichwespecifythedesiredproperties.Inthenextsections,we

willfocusontheformatofthedatabaseandthenatureofconstraints.

Hereisaliner-notelikedescriptionofatypicalmusicprogram.Thepropertiesofthe sequence may be grouped in three categories: 1) user preferences, 2) global

propertiesonthecoherenceofsequences,and3)constraintsontheexploitationofthe


11/16


1999.

11

catalogue.ThefollowingexampledescribesamusicprogramcalledDrivingaCar,

ideallysuitedforlisteningtomusicinacar:

Userpreferences

Note that these constraints specify global properties of the sequence, and do not

specifythepositionofitemsinthesequence:

Noslow/veryslowtempos(CardinalityConstraint)

Atleast30%female-typevoice

Atleast30%purelyinstrumentalpieces

Atleast40%brass

Atmost20%CountryPopstyle

OnesongbyHarryConnickJr.

Constraintsonthecoherenceofthesequence

Stylesoftitlesareclosetotheirneighbors(successorandpredecessor).Thisis to

ensuresomesortofcontinuityinthesequence,style-wise.

Authorsarealldifferent.

Constraintsontheexploitationofthecatalogue

Containstwelvedifferentpieces.ThisistofitonatypicalCDorminidiskformat.

Containsatleast5titlesfromthelabelEpic/SonyMusic.Thisisatypicalbiasto

exploitthecatalogueinaparticularregion.

4.4 DatabaseofMusicTitles

The database required for building music programs contains content information

neededforspecifyingtheconstraints.Moreprecisely,eachitemisdescribedbyaset

ofattributes,whichtaketheirvalueinapredefinedtaxonomy.Theattributesareof

twosorts:technicalattributesandcontentattributes.

Technicalattributesincludethenameofthetitle(e.g.Learntoloveyou),thename

of the author (e.g. Connick Harry Jr.), the duration (e.g. 279 sec), and the

recordinglabel(e.g.Epic/SonyMusic).Contentattributearetypicalmeta-data:they

describemusicalpropertiesofindividualtitles.Theattributesarethefollowing: style

(e.g.JazzCrooner),typeofvoice(e.g.muffled),musicsetup(e.g.instrumental),type of instruments (e.g. brass), tempo (e.g. slow-fast), and other optional

attributessuchasthetypeofmelody(e.g.consonant),orthemainthemeofthelyrics

(e.g.love).

Inthecurrentstateofourproject,thedatabaseiscreatedbyhand,bymusicexperts

(includingthethirdauthor).However,itshouldbenotedthat1)someattributescould

beextractedautomaticallyfromthesignal,suchasthetempo,seee.g.(Scheirer,1998)

and2)alltheattributesaresimple,inthesensethattheydonotrequiresophisticated

musicalanalysistobefilled.

Animportantaspectofthedatabaseis thatthevaluesofcontentattributesarelinked

toeachotherbysimilarityrelations.Thesesimilarityrelationsareusedforspecifyingconstraints on the continuityof the sequence. For instance, the preceding example

containsaconstraintonthecontinuityofstyles.Moregenerally,thetaxonomieson


12/16


1999.

12

attributesvaluesallowtoestablishlinksofpartialsimilaritybetweenitems,according

toaspecificdimensionofmusicalcontent.

Someof theserelations aresimpleordering relations.Forinstancetemposcantake

theirvalueintheorderedlist(fast,fast-slow,slow-fast,slow).Otherattributessuchas

style, take their value in full-fledged taxonomies. The taxonomy of styles is

particularly worthmentioning, because it embodies a global knowledge onmusic

whichisaclearaddedvalueforthesystem.

Various taxonomies ofmusical styles have been designed, particularly by internet

musicretailers,suchasAmazon(1998)orMusicBoulevard(1998).However,these

classifications are mainly designedwith a query-based approach. For example the

taxonomyofstylesproposedbyAmazonisatree-likeclassificationorientedtoward

presentationofitemsinasearch-orientedway.Thistaxonomyembodiesarelationof

generalization/specialization between styles: Blues is more general thanMemphisBlues.Assuch,itiswell suited for navigatinginthe catalogue tofind

under-specifieditems.However,itdoesnotrepresentsimilaritiesbetweenstyles,for

instance,similaritiesbetweenstylesthathavecommonorigins,like,say,Soul-Blues

andJazz-Crooner.

Conversely, we designed a taxonomy of styles representing explicitly relations of

similaritybetweenstyles.Ourtaxonomyisanon-directedgraphinwhichverticesare

stylesandedgesexpresssimilarity.Itcurrentlyincludes120differentstyles,covering

mostofwesternmusic.ApartofthegraphisrepresentedinFigure3.

Soul-JazzLatino-Jazz

Jazz-Swing

Soul-Crooner

Jazz-Crooner

Soul-Funk

Pop-Soul

Soul-Blues

WorldReggae

SoulFunk

Pop-Song Pop-Rock

CountryPop

PopCaliforniaSoul-Crooner

......

...

...

Jazz-CroonerCountry-Crooner

Figure 3. A part of a taxonomy ofmusical styles.Links indicate a similarity

relation between styles. Jazz-Crooner is represented as similar with Soul-

Blues.

4.5 ServicesandInterfaceIssues

Computingmusicprogramsfromadatabaseandasetofconstraintsisshowntobea

complex combinatorial problem.Constraint satisfaction techniques may be used to

solveit,asexplainedin(Pachet&al.,1999).

The resulting technique canbeusedtobuild anumberofservices relatedtomusic

deliverywithlarge-scalemusic catalogues.Welisthereexamplesofcurrentlybuilt

applications:automaticCDassembly,aPathBuilderandaBaroquerecitalcomposer.

Other applications are envisaged for set-top-boxes services and digital audio

broadcastingwhichwedonotdetailhereforreasonsofspace.

SamplerBuilder

Thesimplestapplicationofthistechnologyisasystemtargetedatmusicprofessionals

for building music programs (so-called samplers) from a given database. In the


13/16


1999.

13

application, the user can specify the constraints usingan interface, and launch the

system on a database. This system is aimedat professionals who want to express

explicitlyallthepropertiesofthedesiredprograms,andthushavefullcontrolonalltheconstraints.

Progressiveprograms

Inthisscheme,theuseronlyspecifiesthestylisticstructureoftheprogram:thegenres

ofthe beginning,middle and end.Thismay beused for instancefor creating long

programsforparties,inwhichyouknowinadvancethestructure(e.g.beginwithPop,

thenRock,thenSlows,etc.).

Pathacrossdifferentstyles

Services dedicated to average end users should allow them to express only their

preferences,possiblyusingautomaticprofilingsystems,andcontainpredefined,fixedconstraints for the coherence properties and catalogue exploitation, according to

predeterminedambiencesorconfigurations.Atypicalconfigurationisapathbetween

twotitles.Inthisscheme,theusercanspecifyastartingtitleandanendingtitle.The

systemcontainshiddenconstraintsoncontinuityofgenres,andtemposarefixed.For

instance,findacontinuouspathbetweenClineDionsAllbymyself,andMichael

JacksonsBeatit(seeFigure4).

Figure4.ThePathBuilderprogram.Theuserchoosesastartingandendingtitle,

aswellasadegreeoftightnessbetweensuccessivetitles.

Specificmusicdomains

Theapproachcan beused toproducemusicprogramsinspecificstyles,by adding

domain specific constraints. A prototype application dedicated to Baroque music

implementedinourlaballowstobuildvariousrecitalsinthedomainofBaroque

harpsichordmusic.Baroquemusicisagoodexampleofaspecificdomain,because


14/16


1999.

14

recitals ofBaroquemusic(XVIIth century) follow rules identified bymusicologists

(Bukofzer,1947),whileallowingagreatdealoffreedomtoperformers.Atypicalrule

concerningthestructureofrecitalsisthecontinuityoftemposbetweenconsecutivepieces.Morespecificrulesarealsoused,suchasrulesonthetonality:atthisperiodof

musicalhistory,recitalswhereallowedtomodulate-i.e.changetonality-onlyonce.

Otherconstraintsconcernthestructureoftherecital(introductorypartwithnecessary

piecetypes),aswellasnecessaryalternationofpiecetypes.

The systemallows the user tocreate and listen todifferentmusicprograms,while

ensuringtheconsistencyoftheseprograms,accordingtotherulesofthestructureof

recitals.Thedatabasecontainstitleswithcontentdescriptionadaptedtothedomain.

For instance,attributes suchthattype(e.g.Gigue,Chaconne,etc.), tonality

anddensityareaddedtothedatabasefordescribingrelevantaspectsoftitles.The

constraintsystemcontainstheconstraintscorrespondingtotherulesdescribedabove.

Theresultingsystemallowstoproduceagreatnumberofdifferentrecitals,whichall

havethedesiredpropertiesofgoodrecitals,inthestyleofthecomposerstime(see

Figure5).

Figure 5. The interface forBaroqueComposer. The user can select aBaroque

composer, and then a corresponding catalogue of pieces of Baroque music.

He/she can then build music programs which satisfy the constraints of theBaroquestyle,andlistentothemintherightorder.

Thiskindofserviceliesbetweentwoextremebounds:fixedorderandrandomness.

Ontheonehand,aCDplayedinastandardfashioncontainsafixedmusicprogram.

Ontheotherhand,acommonfeatureofCDplayers(orJukeboxes)istherandom

selection button,whichchooses atrandom betweendifferentCDsandbetween the

titles of the CDs. Constraint techniques provide an intermediary degree of control

betweenthese twoextremes,wheretheusercan stillexpresssomepreferences,but

thesystemcomputesaprogramwhichyieldspropertiesofcoherence.


15/16


1999.

15

4.6 ExplorationSpace

RecitalComposer is an enabling technology for building high-levelmusic deliveryservicesexploitinglarge-scalemusiccatalogues.Thesystemisbasedontheideaof

creatingexplicitsequencesofitems,specifiedbytheirglobalproperties,ratherthan

computing setsofitemssatisfying queries.One ofitsmain advantagesoverquery-

basedorcollaborativefilteringapproachesisthatitproducesreadyforuseordered

sequences ofitems,whichsatisfythe three goals ofmusic selection, i.e. repetition,

surprise,andexploitation ofcatalogues. Itcreatescoherentmusicprogramsfrom

userspecifications,wherethecoherenceisspecifiedintermsofmeta-dataonmusic

titles and assuchcan beseenas anotherexampleofsemanticcontrol,where the

semanticsisthestructureofmusicprograms.Comparedtothejukeboxofthe20s,it

allows to access much larger music catalogues with simple controls (e.. user

preferences)which,onceagain,makesense,withoutrequiringan aprioriknowledgeoftheunderlyingmusiccatalogue.

5. Conclusion

The new landscape ofdigitalmultimedia opensnew doors for interactive listening

environmentswhichprovide richermusical experiences.Wehaveargued thatsuch

environmentsrequiresomesortofsemanticpreservingsystems.Wehaveillustrated

this ideawith two projectscurrently developed atSonyCSL,in the areasofsound

spatialization, and content-basedmusic selection. In both cases, the technology of

constraints is proposed for representing these seeds of semantics, that yield

explorationspaceswithmeaningfulcontrols.Alotremainstobedone,inotherareasof music listening and perception, but these projects already suggest that the

traditional borders between composition, production and listening may have to be

redefined.Inparticular,aquestionwhicharisesiswhatkindofmusiccomposerswill

make,iftheyknowthatlistenershaveactivelisteningdevicesathome?Ifweknow

whattechnologyisintheair,whatmusicwillbeinourears?

References

AmazonMusicStorewebsite,http://www.amazon.com,1998.

Assayag G., Agon C., Fineberg, J., Hanappe P., An Object Oriented VisualEnvironment For Musical Composition, Proceedings of the International

ComputerMusicConference,pp.364-367,Thessaloniki,1997.

Bukofzer,M.MusicintheBaroqueEra,fromMonteverditoBach,W.W.Norton&

Company,1947.

Cavalli-Sforza, L. and Feldman, M. Cultural Transmission and Evolution: a

QuantitativeApproach,PrincetonUniversityPress,1981.

Cruz-Neira,C.,Leight,J.,Papka,M.,Barnes,C.,Cohen,S.M.,Das,S.,Engelmann,

R.,Hudson, R.,Roy, T., Siegel, L., Vasilakis, C.,DeFanti, T.A., Sandin, D.J.,

ScientistsinWonderland: aReportonVisualizationApplications in theCAVE

VirtualRealityEnvironment,Proc. IEEESymp.onResearchFrontiers inV.R.,

pp.59-66,1993.Eco,U.OperaAperta.Bompiani(Milan),1962.


16/16


1999.

16

EckelG.,ExploringMusicalSpacebyMeansofVirtualArchitecture, Proceedings

ofthe8thInternationalSymposiumonElectronicArt,SchooloftheArtInstituteof

Chicago,1997.Epstein,JoshuaM.GrowingArtificialSocieties:SocialSciencefromtheBottomUp ,

MITPress,1996.

Fireflywebsite,http://www.firefly.com,1998.

Infoglidewebsite,http://www.infoglide.com,1998.

Lepain,P.Ecouteinteractivedesdocumentsmusicauxnumriques,inRechercheset

ApplicationsenInformatiqueMusicale ,Chemillier&PachetEds,Hermes,Paris,

1998.

Meyer,L.EmotionsandmeaninginMusic,UniversityofChicagoPress,1956.

Memon,N,Wong,P.W.ProtectingDigitalMediaContent,Communicationsofthe

ACM,July1998,pp.34-43,1998.

MPEG7RequirementsGroup,MPEG-7RequirementsDocument,Doc.ISO/MPEGN2461,MPEGAtlanticCityMeeting,October1998.

MyLaunchwebsite:www.mylaunch.com,1998.

Narmour,E.Theanalysisandcognitionofmelodiccomplexity.UniversityofChicago

Press,1992.

OrlareyY.,FoberD.,LetzS.,Elody:aJava+MidiShareBasedMusicComposition

Environment,ProceedingsoftheICMC,1997.

Pachet, F. Delerue, O. A Constraint-based Temporal Music Spatializer, ACM

MultimediaConference,Brighton(UK),pp.351-360,1998.

Pachet, F.Roy,P.Cazaly,D.ACombinatorialApproachtoContent-basedMusic

Selection,IEEEInternationalConferenceonMultimediaComputingandSystems,

Firenze(Italy),1999.Read, Oliver and Walter Welch. From Tin Foil to Stereo: Evolution of the

Phonograph.Indianapolis,1959,2ndedition1976.

Rosen,C.TheFrontiersofMeaning:ThreeInformalLecturesonMusic,Hill&Wang

Pub,1994.

Shardanand, U. and Maes, P. Social Information Filtering: Algorithms for

Automating "Word ofMouth', Proceedings of the 1995 ACM Conference on

HumanFactorsinComputingSystems,pp.210-217,1995.

Scheirer, E. D. (1998). Tempo and beat analysis of acoustic musical signals,

JournaloftheAcousticalSocietyofAmerica,103(1):588-601.

Sheirer, E. Vnnen, R. Huopaniemi, J. AudioBiffs: The MPEG-4 Standard for

Effects Processing, Proceedings of the first Digital Audio Effects Workshop,Barcelona,pp.159-167,November1998.

Smith,D.Melara,R.Aestheticpreferenceandsyntacticprototypicalityinmusic:Tis

thegifttobesimple,Cognition,34,pp.279-298,1990.

TaubeH.,CommonMusic:AMusicCompositionLanguageinCommonLispand

CLOS,ComputerMusicJournal,vol.15,n2,21-32,1991.

François Pachet, Active Listening : What is in the Air

Documents

Transcript of François Pachet, Active Listening : What is in the Air