Real-Time Corpus-based Concatenative

Synthesis with CataRTDiemo Schwarz

IMTR Real-Time Music Interaction TeamIrcam–Centre Pompidou

http://imtr.ircam.fr CataRT releases

http://ftm.ircam.fr FTM for Max 5

Ircam Overview 24.6.2011

1

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Principle of CBCS

Corpus-basedsounds + segmentation + descriptors (timbre, metadata, classes) concatenative grains + overlap + chainingreal-time synthesis interactive + explorative (navigation)

2

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Motivation and Approachwork with all the nuances of real soundlarge sound databases exist, ready to use

new method → new sound creationdata-driven vs. rule-based approachgeneral superiority of data-driven approaches in many other domains

3

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Influences and LinksInspired by concatenative speech synthesisCBCS in real-time (as implemented in CataRT) can be seen as a content-aware extension to granular synthesis:

providing direct access to specific sound characteristicnot restricted to selection by position in a sound

4

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Applications of CBCS

Interactive exploration of sound databases (browsing)Resynthesis of an audio target (mosaicing)Texture synthesis (ambiences, soundtracks)High-level instrument synthesis (example: Synful)Expressive speech synthesis

5

CataRT — http://imtr.ircam.fr — Diemo Schwarz

A Short History of Data-Driven Synthesis

I.Prehistory (1948)Concrete and early electronic music

II.The Classics — revisited (1980, 1990)SamplingGranular SynthesisJohn Oswald

III.The Modern Times (2000)many research projects, often in mutual ignorance

6

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Tape splicing

soft attack or decay

combined attack and decay of two sounds

medium attack or decay

hard attack or abrupt finish

softer and less abrupt than D

[figures: Mikhail Malt]

Karlheinz Stockhausen: Étude des1000 collants (1952)

7

CataRT — http://imtr.ircam.fr — Diemo Schwarz

John Cage: Williams Mix (1953)

[http://www.medienkunstnetz.de/works/williams-mix]

8

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Related Work up to 2005

1948

19801990

1993

20002001

2001

20012002

2003

2003

2003

2003

2003

20042004

2004

20042005

2005

2005

Selection

Let them sing it for you

Granular

Samplers

Synful

Musescape

CataRT

MoSievius

stochastic SoundscapesSoundclustering

La legende des sieclesN.A.G.

MATConcat

targeted Directed Soundtracks

Soundmosaicing

Input driven resynthesisMPEG−7 Audio Mosaics

Soundspotter

automatic

Ringomatic

Musical Mosaicing

Caterpillar

Audio Analogies

manual segmental high−level

Musique Concretemanual

Plunderphonics

similarity descriptorssimilarityframe spectrum

Analysis

fixed

2005

2003

[Schwarz, JNMR 2006]

9

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Current Related WorkTimbreID (William Brent) is a collection of PD externals that allow descriptor analysis, corpus-based concatenative synthesis and CataRT-style 2D interactionExtending voice-driven synthesis to audio mosaicing (Jordi Janer, Maarten de Boer): a system for controlling audio mosaicing with a voice signalMused: Navigating the Personal Sample Library (Graham Coleman): a system for sample-based music compositionScrambledHacks (2006): performance system for spectral lookup alongside ganged music video concatenation

10

CataRT — http://imtr.ircam.fr — Diemo Schwarz

CataRT Introductionflexible testbed to experiment real-time corpus-based concatenative synthesis: CataRTrealisation of CBCS in real-time

patch for Max/MSP with FTM, Gabor, MnM by Norbert Schnell et al. from http://ftm.ircam.frreleased as Free Software under GPL at http://imtr.ircam.fr mailing list concat on http://list.ircam.frmodular MVC architecture:

11

CatArt (?)

CataRT

12

CataRT — http://imtr.ircam.fr — Diemo Schwarz

CataRT Overview

CorpusAnalysis

DB

Synthesis

Audio Input Control Input

13

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Analysis

Corpus

Soundfile+markers

+descriptors

Persistent Sound and Unit

Database

descriptor analysis

Soundfile+grain sizeor markers

RT audio input

Audio Input

onset detection

Soundfile

14

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Synthesis

Corpusnearest

neighbour selection

Target descriptor values Control

transformation + concatenation

descriptor analysis

RT audio input

onset detection

controller

15

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Architectureone catart.data per corpusone catart.import for segmentation and analysisany number of catart.data.proxy to access dataany number of catart.lcd for display and controlany number of catart.selection per corpusany number of catart.synthesis per selection or per corpus

16

Composers and Projects using CataRTSound design and compositionLuis Naõn, Hans Tutschku, Matthew Burtner, Sebastien Roux, Hector Parra, Roque Rivas, Ben Hackbarth

InstallationPlumage (project Enigmes about the navigable score, Roland Cahen), Pierre Jodlowski, Cécile Babiole, Franck Leibovici

Live electro-acoustic music Dai Fujikura, Stefano Gervasoni, Miguel Angel Ortiz-Pérez, Sam Britton, Aaron Einbond, Bruno Ruviaro, Christopher Trapani, Marc Vitoria, Eric Maestri, theconcatenator

Examples: myspace.com/catartsoftware

!"

!"

18

""

""

q = 60

q = 60

!"

!"

q = 102

q = 102

!#

!#

""

""

!"

!"

A. Fl.(smallgroup)

B. Cl.(smallgroup)

Tpt.(smallgroup)

1 Vln.(smallgroup)

1 Vc.(smallgroup)

electronics

Fl.

Ob. I

Ob. II

Cl. I

Cl. II

Bsn. I

Bsn. II

Hn. I

Hn. II

Hn. III

Hn. IV

Tpt.

Tbn. I

Tbn. II

Timp.

B. D.

Vln. I

Vln. II

Vla.

Vc.

Db.

!

"

!

"

!

"

!

"

!

"

!

"

!"

$SMALL GROUP

%&'%( % % %) % # % #

p

* + + ,ca. 20 - 21 sec.

, ,- , , , ,

$ % %( %&'(% % % #

p

%) * , , , ,- , , , ,

$ % % #

p

% #) . + , , , ,- , , , ,

$ %&'( #% % #

% % %p

+ / , , ,- , , , ,

0 $% %1 % % %p

% #) . , , , ,- , , , ,

$

section 3

*

4

%%%%%%%%%(

211( %%%%%%%

((

more and more dense and violent(grain size gets smaller and smaller)Harmony 3Harmony 2

gliss.

gliss.

+3 semitones,

-4 semitones,

%%%%%%%%%1

2222 %%%%%%%22(1 ,

CUT OFF

5

34 5 . * + /

$ , , , , ,- +marcato

*ff

$%'6 $%

'6$%

'6%7( %7($%

'6

%7 . . %7 . $%'

( 6$%

'6%7( %7 .$%

'6 %78 . * $%'6%78 . * .

$%'

6$%

'( 6

%7 .%7

*sffffz

$%'6%7'( % * * espress.

fff

. $%'6 %8 %( %

$%'6

%% #

$ , , , , ,- +marcato

*ff

$%'( 6 $%'

( 6%7 %71 * $%

'6 %78 . + $%'( 6 $%'6

%7 .%7(

.$%

'6$%'6%7 %7 * %

%')$%

'( 6

$%'6

%7 %7

. %7( *sffffz

%7'(8 . *

espress.

*fff

%) $%'6% %( # % %

$ , , , , ,- + marcato*ff

. %%'()

$%'6

%74 $%'6%7(

. . %7 .

$%'

( 6 %78 *$%

'6 %7(8 . * * $%'6

$%'6

%7 %7 . $%

'( 6 %7(8 * .

$%'6

sffffz

%78fff

%7'(. . % %8 . * + *

espress.

%6

$ , , , , ,- +marcato

*ff

%%') %74 .

%%

'

2 )%%')

$%'6

%7%7 .

%7

(.

%%')

%%

'

)%%

'

)%7( %7 %7* $%'( 6 $%

'6

%%

'

)

%7 %7(%%')

%%

'

)%7. %7 %7

. %%')

%%

'

)$%

'6

%7

( .%7( %7 .

sffffz fff

$%

'

( 6

%7'

(

%(. + *

espress.

%%')

%%(

%%'

()%

%

$ORCHESTRA

, , , , ,- +marcato

* .ff

%%')

$%'( 6

%74%%

'

2 )%7 .

%7. .

%%

'

) %%')%7 %7 . %

%')%%

'

) %%')

$%'

( 6

%7 %72. %7

$%'

( 6

%%')

%7( . . %71%%

'

)%72. %7

.%%

'

)%%')

sffffz

%7 %7fff

espress.

%%'

()$

%

'

(6

%7'% %( %

% #%

% #%

%%

'

)%

% %

9 , , , , ,- +marcato

*ff

$%'2 6%78 .$%

'6

$%'6%7 . %7 .

$%'

6$%

'6$%'2 6 $%

'6%7 %72 . %7 $%'6 $%'6

%%')

%7 %7 . %7%%')

$%'6

%7 . . %7$%

'6%7 .%7

.$%

'6 %78 .

sffffz fff

$%'

( 6 espress.

%7'

2%

%( . $%'

( 6%( %( % %) *

$%'6 % # %

9 , , , , ,- +marcato

*ff

%%') %74 . .

$%'6

$%'6%7

%7 . %

%'2) %%') $%

'6%7 %7 . %7 * %

%') %%')

$%'6

%7 %7$%

'6%7 . %7 . . %%') %74 .

$%'

6 %78 *sffffz

espress.

fff

%7' % %( % %

) * $%'

( 6 %( # % %) *

$ , , , , ,- + + marcato

ff

. $%'6

%74 * 9 . $%'6

$%'

2 6%7. %7 . $%

'6 $%'6%7 . %7 * . $%'6%78 * $%

'6 %78 . *sffffz

ff

$%'

( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *

$ , , , , ,- + 9+ marcato*ff

$%'2 6 %78 . .$%

'6 %728 *$%

'6%78 . *$%

'6%78 . * $%'6 %78 . * $%

'6 %728 .sffffz ff

$%'6%7'

. . %( . $%'( 6 %78 * $%'6

$%'

( 6%7 . . %7 + $

$ , , , , ,- + marcato9 *ff

$%'6%78 . .

$%'6%78 * * . $%

'6$%

'6%728 $%'6%7 . . %7 + * $%'

( 6 %78 . * $sffffz

%7'8 . *ff

*$%

'6 %7( 4 . * *

$%'

6%74 . * 9

$ , , , , ,- + 9*ff

marcato$%'2 6%78 . * $%'6%78 . . $%

'6%74 * * . %

%') %74 * . $%'6

$%'6

%74 %728 . * .$%

'6

sffffz

%78ff

%%')%7'( %7( * * $%

'( 6 %78 . * $ . $%'6 %7( 4 *

%%'

() %7( 4 .

$ , , , , ,- + + marcato

ff

$%'( 6%78 . * *$%

'6 %78 . .

$%

'6 %%

')%72

. %7. $%'( 6%78 * . $%

'6 %7(8 * *sffffz ff

$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *

9 , , , , ,- , , ,(senza sord.)

sffffz

%7'2 4 . * + /

9 , , , , ,- , , ,straight mute

sffffz

%7'(8 . *super short staccato

+p

%7'( 7' 7' 7' 7'

5

gliss.

9 , , , , ,- , , ,p

soft sticks

%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3

gliss.

: , , , , ,- , , ,sffffz

%7'4 . * + /

$ , , , , ,- marcato+ *ff

$%'6 %7(8 .

$%'6

$%'

6%7( . . %7 * $%

'6 %7(8 .$%

'6 $%'6%7(

. .%7

* $%'6

%74 . . $%'

( 6 %7(8 *$%

'6 %74 .

sffffz

unis.

fff

$%'6 espress.%7'( %

%($%'6

% %% %( % %4 . *

$ , , , , ,-div.

marcato+ * .ff

$%'6 %7(8 * $%

'6 %78 . . $%'( 6 %7(8 * *

$%'6 %78 .

$%'6 $%

'( 6%7 . . %71 . $%

'( 6 $%

'6%%')%7(

.%7

. %7

$ , , , , ,-

div.

marcato

+ *ff

$%'( 6%78 .$%

'6$%

'6%7(. .

%7(*

$%

'6 %74 . %

%'()%%')

%7

( . . %7* . %

%') %78 .$%

'( 6

$%'

6sffffz

unis.%7* .

%7

2fff

$%'6 espress.

$%'

6%7'( %7 . %(

%) * /

$ , , , , ,-marcato

+ff

* .$%

'( 6

%718 * . $%'6 %7(8 .

$%'

( 6 %78 *

$%'6

$%'6%7( . %7 .

$%'6

$%'

6%7

. %7 .

$%'

6 %%')%72

. . %7

( *

0 , , , , ,-div.

marcato+ *ff

$%6%74 . $%

'2 6 %

%')%7 . . %7 * $%

'6 %78 . . $%'6 %72 4 *

$%'

( 6$%

'6%7 . %7 . %

%') %78 . $%'

2 6%74 .

$%'

6 %74 .

sffffz

unis.

fff

espress.%7' %( %$%

'6%

%( % %% #

% % *

0 , , , , ,- marcato+ff

* .$%( 6

%78 * $%'

2 6%74 . . $%'6 %72 4 *

$%'

( 6 %78 . *$%

'6$%

'6%7 . %7 . . $%

'6%78 * . %%') %72 4

9 , , , , ,- +marcato

*ff

$%'6

%74 . .$%

'6 %78 * $%'2 6$%

'6%7 . . %7 * $%

'6%74 . $%'6

%%')%7 . . %7 * .

$%'

6 %78 *

sffffz

unis.

%%'()

espress.

%7'%( .

fff

. %%')

% %($%

'( 6

% % % +

9 , , , , ,-div.

+marcato

*ff

.$%

'6%78 * $%

'6%74 . . $%

'6 %78 *$%

'6

$%'6%7

. %7 . $%'2 6

%%')%7 . . %7 +

$%'

6 %78 .

9 , , , , ,- +marcato

*ff

$%'6

%78 . $%'6$%

'6%7. %7 . .

$%'6

%78 * .$%

'2 6 %78 *

$%'6

%78 . * .$%

'6%78 +

sffffz

%7'%7( * *

fff

$%'6 %(

) + * %)

2

17

Installation GrainStick“Rainstick” gestural control metaphor

Demo of SAME (Sound And Music for Everyone Everyday Everywhere Every way) EU project

Composition of sounds and ambiences by Pierre Jodlowski

WFS spatialisation and Motion Tracking by Grace Leslie and the Room Acoustics team

Gesture analysis by Bruno Zamborlin and Diemo Schwarz

18

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Grainstick Example 1

19

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Grainstick Example 2

20

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Grainstick InstallationsPrototype: Agora Festival 2009, Ircam, ParisPublic installation: 6. June 2010, Cité des Sciences, Paris

including video projection, more collaborative elements, several universes

Supports et concept de diffusion

Structure tubulaire démontable autonome de 10 mètres de diamètre. Surface de projection en

toile de cinéma circulaire et système de diffusions vidéo et audio intégrées. Ouverture 1/6e de

la circonférence pour l'entrée du public.

21

Installation Xe-RocksTurning photocopiers into musical instruments, by video and sound artist Cécile Babiole

On show at the ECM Gantner, Bourogne

22

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Installation PlumageCollaboration with national design school ENSCI led by Roland Cahen, Yoan Ollivier, Benjamin Wulf (ENSCI), Christian Jacquemin, Rami Ajaj (LIMSI), Diemo Schwarz (Ircam)

within the project ENIGMES about the navigable score3D interface using Virtual Choreographer http://virchor.sourceforge.net

23

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Plumage

24

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Plumage

25

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Plumage

26

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Live Recording the CorpusCorpus-based improvisation with live sound input

Diemo Schwarz with Etienne Brunet

Live Algorithms for Music (LAM) concerts:Improvisation of George Lewis, Evan Parker, Sam Britton, Diemo Schwarz and othersRien du tout by Sam Britton, Diemo Schwarz

27

28

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Electro–Acoustic Music (1)

Whisper Not by Stefano Gervasoni for viola and electronics, computer music realization by Thomas Goepfer

“response” of CataRT to the instrumentmorphing between corpora of pizzicato viola and water-drops

Interaction with pre-recorded sound,

interpolation, rearranging, navigation

29

!"

!"

18

""

""

q = 60

q = 60

!"

!"

q = 102

q = 102

!#

!#

""

""

!"

!"

A. Fl.(smallgroup)

B. Cl.(smallgroup)

Tpt.(smallgroup)

1 Vln.(smallgroup)

1 Vc.(smallgroup)

electronics

Fl.

Ob. I

Ob. II

Cl. I

Cl. II

Bsn. I

Bsn. II

Hn. I

Hn. II

Hn. III

Hn. IV

Tpt.

Tbn. I

Tbn. II

Timp.

B. D.

Vln. I

Vln. II

Vla.

Vc.

Db.

!

"

!

"

!

"

!

"

!

"

!

"

!"

$SMALL GROUP

%&'%( % % %) % # % #

p

* + + ,ca. 20 - 21 sec.

, ,- , , , ,

$ % %( %&'(% % % #

p

%) * , , , ,- , , , ,

$ % % #

p

% #) . + , , , ,- , , , ,

$ %&'( #% % #

% % %p

+ / , , ,- , , , ,

0 $% %1 % % %p

% #) . , , , ,- , , , ,

$

section 3

*

4

%%%%%%%%%(

211( %%%%%%%

((

more and more dense and violent(grain size gets smaller and smaller)Harmony 3Harmony 2

gliss.

gliss.

+3 semitones,

-4 semitones,

%%%%%%%%%1

2222 %%%%%%%22(1 ,

CUT OFF

5

34 5 . * + /

$ , , , , ,- +marcato

*ff

$%'6 $%

'6$%

'6%7( %7($%

'6

%7 . . %7 . $%'

( 6$%

'6%7( %7 .$%

'6 %78 . * $%'6%78 . * .

$%'

6$%

'( 6

%7 .%7

*sffffz

$%'6%7'( % * * espress.

fff

. $%'6 %8 %( %

$%'6

%% #

$ , , , , ,- +marcato

*ff

$%'( 6 $%'

( 6%7 %71 * $%

'6 %78 . + $%'( 6 $%'6

%7 .%7(

.$%

'6$%'6%7 %7 * %

%')$%

'( 6

$%'6

%7 %7

. %7( *sffffz

%7'(8 . *

espress.

*fff

%) $%'6% %( # % %

$ , , , , ,- + marcato*ff

. %%'()

$%'6

%74 $%'6%7(

. . %7 .

$%'

( 6 %78 *$%

'6 %7(8 . * * $%'6

$%'6

%7 %7 . $%

'( 6 %7(8 * .

$%'6

sffffz

%78fff

%7'(. . % %8 . * + *

espress.

%6

$ , , , , ,- +marcato

*ff

%%') %74 .

%%

'

2 )%%')

$%'6

%7%7 .

%7

(.

%%')

%%

'

)%%

'

)%7( %7 %7* $%'( 6 $%

'6

%%

'

)

%7 %7(%%')

%%

'

)%7. %7 %7

. %%')

%%

'

)$%

'6

%7

( .%7( %7 .

sffffz fff

$%

'

( 6

%7'

(

%(. + *

espress.

%%')

%%(

%%'

()%

%

$ORCHESTRA

, , , , ,- +marcato

* .ff

%%')

$%'( 6

%74%%

'

2 )%7 .

%7. .

%%

'

) %%')%7 %7 . %

%')%%

'

) %%')

$%'

( 6

%7 %72. %7

$%'

( 6

%%')

%7( . . %71%%

'

)%72. %7

.%%

'

)%%')

sffffz

%7 %7fff

espress.

%%'

()$

%

'

(6

%7'% %( %

% #%

% #%

%%

'

)%

% %

9 , , , , ,- +marcato

*ff

$%'2 6%78 .$%

'6

$%'6%7 . %7 .

$%'

6$%

'6$%'2 6 $%

'6%7 %72 . %7 $%'6 $%'6

%%')

%7 %7 . %7%%')

$%'6

%7 . . %7$%

'6%7 .%7

.$%

'6 %78 .

sffffz fff

$%'

( 6 espress.

%7'

2%

%( . $%'

( 6%( %( % %) *

$%'6 % # %

9 , , , , ,- +marcato

*ff

%%') %74 . .

$%'6

$%'6%7

%7 . %

%'2) %%') $%

'6%7 %7 . %7 * %

%') %%')

$%'6

%7 %7$%

'6%7 . %7 . . %%') %74 .

$%'

6 %78 *sffffz

espress.

fff

%7' % %( % %

) * $%'

( 6 %( # % %) *

$ , , , , ,- + + marcato

ff

. $%'6

%74 * 9 . $%'6

$%'

2 6%7. %7 . $%

'6 $%'6%7 . %7 * . $%'6%78 * $%

'6 %78 . *sffffz

ff

$%'

( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *

$ , , , , ,- + 9+ marcato*ff

$%'2 6 %78 . .$%

'6 %728 *$%

'6%78 . *$%

'6%78 . * $%'6 %78 . * $%

'6 %728 .sffffz ff

$%'6%7'

. . %( . $%'( 6 %78 * $%'6

$%'

( 6%7 . . %7 + $

$ , , , , ,- + marcato9 *ff

$%'6%78 . .

$%'6%78 * * . $%

'6$%

'6%728 $%'6%7 . . %7 + * $%'

( 6 %78 . * $sffffz

%7'8 . *ff

*$%

'6 %7( 4 . * *

$%'

6%74 . * 9

$ , , , , ,- + 9*ff

marcato$%'2 6%78 . * $%'6%78 . . $%

'6%74 * * . %

%') %74 * . $%'6

$%'6

%74 %728 . * .$%

'6

sffffz

%78ff

%%')%7'( %7( * * $%

'( 6 %78 . * $ . $%'6 %7( 4 *

%%'

() %7( 4 .

$ , , , , ,- + + marcato

ff

$%'( 6%78 . * *$%

'6 %78 . .

$%

'6 %%

')%72

. %7. $%'( 6%78 * . $%

'6 %7(8 * *sffffz ff

$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *

9 , , , , ,- , , ,(senza sord.)

sffffz

%7'2 4 . * + /

9 , , , , ,- , , ,straight mute

sffffz

%7'(8 . *super short staccato

+p

%7'( 7' 7' 7' 7'

5

gliss.

9 , , , , ,- , , ,p

soft sticks

%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3

gliss.

: , , , , ,- , , ,sffffz

%7'4 . * + /

$ , , , , ,- marcato+ *ff

$%'6 %7(8 .

$%'6

$%'

6%7( . . %7 * $%

'6 %7(8 .$%

'6 $%'6%7(

. .%7

* $%'6

%74 . . $%'

( 6 %7(8 *$%

'6 %74 .

sffffz

unis.

fff

$%'6 espress.%7'( %

%($%'6

% %% %( % %4 . *

$ , , , , ,-div.

marcato+ * .ff

$%'6 %7(8 * $%

'6 %78 . . $%'( 6 %7(8 * *

$%'6 %78 .

$%'6 $%

'( 6%7 . . %71 . $%

'( 6 $%

'6%%')%7(

.%7

. %7

$ , , , , ,-

div.

marcato

+ *ff

$%'( 6%78 .$%

'6$%

'6%7(. .

%7(*

$%

'6 %74 . %

%'()%%')

%7

( . . %7* . %

%') %78 .$%

'( 6

$%'

6sffffz

unis.%7* .

%7

2fff

$%'6 espress.

$%'

6%7'( %7 . %(

%) * /

$ , , , , ,-marcato

+ff

* .$%

'( 6

%718 * . $%'6 %7(8 .

$%'

( 6 %78 *

$%'6

$%'6%7( . %7 .

$%'6

$%'

6%7

. %7 .

$%'

6 %%')%72

. . %7

( *

0 , , , , ,-div.

marcato+ *ff

$%6%74 . $%

'2 6 %

%')%7 . . %7 * $%

'6 %78 . . $%'6 %72 4 *

$%'

( 6$%

'6%7 . %7 . %

%') %78 . $%'

2 6%74 .

$%'

6 %74 .

sffffz

unis.

fff

espress.%7' %( %$%

'6%

%( % %% #

% % *

0 , , , , ,- marcato+ff

* .$%( 6

%78 * $%'

2 6%74 . . $%'6 %72 4 *

$%'

( 6 %78 . *$%

'6$%

'6%7 . %7 . . $%

'6%78 * . %%') %72 4

9 , , , , ,- +marcato

*ff

$%'6

%74 . .$%

'6 %78 * $%'2 6$%

'6%7 . . %7 * $%

'6%74 . $%'6

%%')%7 . . %7 * .

$%'

6 %78 *

sffffz

unis.

%%'()

espress.

%7'%( .

fff

. %%')

% %($%

'( 6

% % % +

9 , , , , ,-div.

+marcato

*ff

.$%

'6%78 * $%

'6%74 . . $%

'6 %78 *$%

'6

$%'6%7

. %7 . $%'2 6

%%')%7 . . %7 +

$%'

6 %78 .

9 , , , , ,- +marcato

*ff

$%'6

%78 . $%'6$%

'6%7. %7 . .

$%'6

%78 * .$%

'2 6 %78 *

$%'6

%78 . * .$%

'6%78 +

sffffz

%7'%7( * *

fff

$%'6 %(

) + * %)

2

Electro–Acoustic Music (2)

swarming essence by Dai Fujikura for orchestra and electronics, computer music realization by Manuel Poletti

harmonic selection of specially composed small instrument group and extended playing style

corpus-based orchestration, separate treatment

Re-orchestration, rearranging, navigation

30

!"

!"

18

""

""

q = 60

q = 60

!"

!"

q = 102

q = 102

!#

!#

""

""

!"

!"

A. Fl.(smallgroup)

B. Cl.(smallgroup)

Tpt.(smallgroup)

1 Vln.(smallgroup)

1 Vc.(smallgroup)

electronics

Fl.

Ob. I

Ob. II

Cl. I

Cl. II

Bsn. I

Bsn. II

Hn. I

Hn. II

Hn. III

Hn. IV

Tpt.

Tbn. I

Tbn. II

Timp.

B. D.

Vln. I

Vln. II

Vla.

Vc.

Db.

!

"

!

"

!

"

!

"

!

"

!

"

!"

$SMALL GROUP

%&'%( % % %) % # % #

p

* + + ,ca. 20 - 21 sec.

, ,- , , , ,

$ % %( %&'(% % % #

p

%) * , , , ,- , , , ,

$ % % #

p

% #) . + , , , ,- , , , ,

$ %&'( #% % #

% % %p

+ / , , ,- , , , ,

0 $% %1 % % %p

% #) . , , , ,- , , , ,

$

section 3

*

4

%%%%%%%%%(

211( %%%%%%%

((

more and more dense and violent(grain size gets smaller and smaller)Harmony 3Harmony 2

gliss.

gliss.

+3 semitones,

-4 semitones,

%%%%%%%%%1

2222 %%%%%%%22(1 ,

CUT OFF

5

34 5 . * + /

$ , , , , ,- +marcato

*ff

$%'6 $%

'6$%

'6%7( %7($%

'6

%7 . . %7 . $%'

( 6$%

'6%7( %7 .$%

'6 %78 . * $%'6%78 . * .

$%'

6$%

'( 6

%7 .%7

*sffffz

$%'6%7'( % * * espress.

fff

. $%'6 %8 %( %

$%'6

%% #

$ , , , , ,- +marcato

*ff

$%'( 6 $%'

( 6%7 %71 * $%

'6 %78 . + $%'( 6 $%'6

%7 .%7(

.$%

'6$%'6%7 %7 * %

%')$%

'( 6

$%'6

%7 %7

. %7( *sffffz

%7'(8 . *

espress.

*fff

%) $%'6% %( # % %

$ , , , , ,- + marcato*ff

. %%'()

$%'6

%74 $%'6%7(

. . %7 .

$%'

( 6 %78 *$%

'6 %7(8 . * * $%'6

$%'6

%7 %7 . $%

'( 6 %7(8 * .

$%'6

sffffz

%78fff

%7'(. . % %8 . * + *

espress.

%6

$ , , , , ,- +marcato

*ff

%%') %74 .

%%

'

2 )%%')

$%'6

%7%7 .

%7

(.

%%')

%%

'

)%%

'

)%7( %7 %7* $%'( 6 $%

'6

%%

'

)

%7 %7(%%')

%%

'

)%7. %7 %7

. %%')

%%

'

)$%

'6

%7

( .%7( %7 .

sffffz fff

$%

'

( 6

%7'

(

%(. + *

espress.

%%')

%%(

%%'

()%

%

$ORCHESTRA

, , , , ,- +marcato

* .ff

%%')

$%'( 6

%74%%

'

2 )%7 .

%7. .

%%

'

) %%')%7 %7 . %

%')%%

'

) %%')

$%'

( 6

%7 %72. %7

$%'

( 6

%%')

%7( . . %71%%

'

)%72. %7

.%%

'

)%%')

sffffz

%7 %7fff

espress.

%%'

()$

%

'

(6

%7'% %( %

% #%

% #%

%%

'

)%

% %

9 , , , , ,- +marcato

*ff

$%'2 6%78 .$%

'6

$%'6%7 . %7 .

$%'

6$%

'6$%'2 6 $%

'6%7 %72 . %7 $%'6 $%'6

%%')

%7 %7 . %7%%')

$%'6

%7 . . %7$%

'6%7 .%7

.$%

'6 %78 .

sffffz fff

$%'

( 6 espress.

%7'

2%

%( . $%'

( 6%( %( % %) *

$%'6 % # %

9 , , , , ,- +marcato

*ff

%%') %74 . .

$%'6

$%'6%7

%7 . %

%'2) %%') $%

'6%7 %7 . %7 * %

%') %%')

$%'6

%7 %7$%

'6%7 . %7 . . %%') %74 .

$%'

6 %78 *sffffz

espress.

fff

%7' % %( % %

) * $%'

( 6 %( # % %) *

$ , , , , ,- + + marcato

ff

. $%'6

%74 * 9 . $%'6

$%'

2 6%7. %7 . $%

'6 $%'6%7 . %7 * . $%'6%78 * $%

'6 %78 . *sffffz

ff

$%'

( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *

$ , , , , ,- + 9+ marcato*ff

$%'2 6 %78 . .$%

'6 %728 *$%

'6%78 . *$%

'6%78 . * $%'6 %78 . * $%

'6 %728 .sffffz ff

$%'6%7'

. . %( . $%'( 6 %78 * $%'6

$%'

( 6%7 . . %7 + $

$ , , , , ,- + marcato9 *ff

$%'6%78 . .

$%'6%78 * * . $%

'6$%

'6%728 $%'6%7 . . %7 + * $%'

( 6 %78 . * $sffffz

%7'8 . *ff

*$%

'6 %7( 4 . * *

$%'

6%74 . * 9

$ , , , , ,- + 9*ff

marcato$%'2 6%78 . * $%'6%78 . . $%

'6%74 * * . %

%') %74 * . $%'6

$%'6

%74 %728 . * .$%

'6

sffffz

%78ff

%%')%7'( %7( * * $%

'( 6 %78 . * $ . $%'6 %7( 4 *

%%'

() %7( 4 .

$ , , , , ,- + + marcato

ff

$%'( 6%78 . * *$%

'6 %78 . .

$%

'6 %%

')%72

. %7. $%'( 6%78 * . $%

'6 %7(8 * *sffffz ff

$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *

9 , , , , ,- , , ,(senza sord.)

sffffz

%7'2 4 . * + /

9 , , , , ,- , , ,straight mute

sffffz

%7'(8 . *super short staccato

+p

%7'( 7' 7' 7' 7'

5

gliss.

9 , , , , ,- , , ,p

soft sticks

%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3

gliss.

: , , , , ,- , , ,sffffz

%7'4 . * + /

$ , , , , ,- marcato+ *ff

$%'6 %7(8 .

$%'6

$%'

6%7( . . %7 * $%

'6 %7(8 .$%

'6 $%'6%7(

. .%7

* $%'6

%74 . . $%'

( 6 %7(8 *$%

'6 %74 .

sffffz

unis.

fff

$%'6 espress.%7'( %

%($%'6

% %% %( % %4 . *

$ , , , , ,-div.

marcato+ * .ff

$%'6 %7(8 * $%

'6 %78 . . $%'( 6 %7(8 * *

$%'6 %78 .

$%'6 $%

'( 6%7 . . %71 . $%

'( 6 $%

'6%%')%7(

.%7

. %7

$ , , , , ,-

div.

marcato

+ *ff

$%'( 6%78 .$%

'6$%

'6%7(. .

%7(*

$%

'6 %74 . %

%'()%%')

%7

( . . %7* . %

%') %78 .$%

'( 6

$%'

6sffffz

unis.%7* .

%7

2fff

$%'6 espress.

$%'

6%7'( %7 . %(

%) * /

$ , , , , ,-marcato

+ff

* .$%

'( 6

%718 * . $%'6 %7(8 .

$%'

( 6 %78 *

$%'6

$%'6%7( . %7 .

$%'6

$%'

6%7

. %7 .

$%'

6 %%')%72

. . %7

( *

0 , , , , ,-div.

marcato+ *ff

$%6%74 . $%

'2 6 %

%')%7 . . %7 * $%

'6 %78 . . $%'6 %72 4 *

$%'

( 6$%

'6%7 . %7 . %

%') %78 . $%'

2 6%74 .

$%'

6 %74 .

sffffz

unis.

fff

espress.%7' %( %$%

'6%

%( % %% #

% % *

0 , , , , ,- marcato+ff

* .$%( 6

%78 * $%'

2 6%74 . . $%'6 %72 4 *

$%'

( 6 %78 . *$%

'6$%

'6%7 . %7 . . $%

'6%78 * . %%') %72 4

9 , , , , ,- +marcato

*ff

$%'6

%74 . .$%

'6 %78 * $%'2 6$%

'6%7 . . %7 * $%

'6%74 . $%'6

%%')%7 . . %7 * .

$%'

6 %78 *

sffffz

unis.

%%'()

espress.

%7'%( .

fff

. %%')

% %($%

'( 6

% % % +

9 , , , , ,-div.

+marcato

*ff

.$%

'6%78 * $%

'6%74 . . $%

'6 %78 *$%

'6

$%'6%7

. %7 . $%'2 6

%%')%7 . . %7 +

$%'

6 %78 .

9 , , , , ,- +marcato

*ff

$%'6

%78 . $%'6$%

'6%7. %7 . .

$%'6

%78 * .$%

'2 6 %78 *

$%'6

%78 . * .$%

'6%78 +

sffffz

%7'%7( * *

fff

$%'6 %(

) + * %)

2

31

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Live Control and Composition Beside Oneself, Aaron Einbond, 2008 for viola and electronics:

real time control of CataRT from audio analysis of the viola

What the blind see 2009 for small ensemble and electronics

real time control of CataRT from audio analysis of the viola

corpus-based “orchestration” or transcription

texture example:1. Rain on leaves

2. CataRT resynthesis with extended technique instrument sounds

32

Aaron Einbond – What the blind see

B

&

&

ã

ã

t

&

ã

Alto

Cl.B.en Sib

Hp.

Pno.

Vib.

F

◊

√ √ √

C10

C10

C10

a tempo

a tempo

a tempo

1936

T T .Tπ

Œ Ù .T

∏T

∑6

≈ . ym

bruit des clefs : doigtés ad lib.

"f "yn y

7

TmflM

ßym y ym ym y Ù TM

flßy ym y y yb

7

® x xx x xm

œSF neige to concatenation

Œ

7

‰ ® jœb œbp

œœ œ œm œ œ

7

‰ œ œ> œ œ .œ

π≈

œétouffé avec papier sempre

p

œm

Ó

Ó ?

Œ ‰ jœb˝

‰ ≈6

˝superball

Vib.

p

3‰6˝ ‰

° sempre

ongles

∑

∑

6

® Tn -

pT JT Tb T

6

ym"f "

ym ym yy T- JT

∑

Jœ

≈

6 6 6 6 6>paume

ß

6

œ

pœ œ œm œb

œ œ

6æ j6æ

jyO>߉ 5≈ .

ym>

pplectre

y

˝

> . .y

yO

>

3

‰ œn ˝

® .

.œb˝

6≈̋

6

œn ˝

≈ 6˝

.6˝

7

7

7‰ ® yæ gliss.(main)

arco

πIVyæ 3

yb y. y. y. y.∑

6

Tym

ym y y y

6

yn ym ym yn ymT- JT Tb T .TbflM

ßTb Mflß

∑

6

® œm œ

.œœ œb œ œ

œ .œ>7

œ .œ œ œ

7

‰ ® œ œb7

∑

jyO

7

.yO

yb

π̋y

6

‰ ®6

6

6˝

œb ˝

6. ‰6

®œ˝

6>ß

28

1. snow melting on a metal roof 2. CataRT resynthesis with instrument sounds3. manually edited transcription 4. live reading by ensemble

33

Bruno Ruviaro – Intellectual Improperty 0.6

Using Catork, a CataRT-based system for electronic music composition for laptop orchestrashttp://www.brunoruviaro.com/catork

Premiered by the Stanford Laptop Orchestra (SLOrk) April, 2010 (chamber version with six laptops), and June (full ensemble version)

34

Gestural Control

35

CataRT — http://imtr.ircam.fr — Diemo Schwarz

New Concepts for Musical Applications

Re-arranging units by other rules than temporal order

Composition = Navigation through the sound space of heterogeneous sound databases

exploit richness of detail of recorded sound

efficient control by perceptually and musically meaningful descriptors

Interaction with self-recorded sound: sound of a musician is available for interaction beyond simple repetition

Cross-selection and interpolation: apply sound characteristics from one corpus to another, morphing between corpora

Orchestration and re-orchestration: matching a mass of sounds to a harmonic or sonic target while retaining precise control over the result

36

Current and Future DirectionsInterface

multi-dimensional scaling by mass–spring–damper model

uniformisation of density

Sound texture synthesis

Modeling of transitions and articulations

40 60 80

0.05

0.1

0.15

0.2

0.25

light rain

40 60 80

0.05

0.1

0.15

0.2

0.25

medium rain

40 60 80

0.05

0.1

0.15

0.2

0.25

heavy rain

37

CataRT — http://imtr.ircam.fr — Diemo Schwarz

AcknowledgementsCataRT is based on FTM&Co by Norbert Schnell et al.Thanks go to:

Alexis Baskind, Julien Bloit, Greg Beller, Miguel Angel Ortiz PérezRoland Cahen, Yoan Olliver, Benjamin Wulf (ENSCI),Christian Jacquemin, Rami Ajaj (LIMSI)

Norbert Schnell, Riccardo Borghesi, Fréderic Bevilacqua, Rémy Muller, Jean-Philippe Lambert (IMTR Team)the French National Agency of Research ANR and the projects Sample Orchestrator and Topophonie

Links:CataRT wiki on imtr.ircam.fr,

mailinglist concat@ircam.fr and ftm@ircam.fr on http://lists.ircam.fr

38

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Exercise SuggestionsAdapting CataRT

Integration in your concert patchAdapting to your musical needs/ideas

Improving CataRTCorpus Data

soundfile editor (list/delete sounds in corpus)

soundset editor (define sound sets, add/remove units)

descriptor range viewer/editor (e.g.: exclude units below threshold)

Analysissegmentation editor

Selectionconstraints: don’t repeat unit

Synthesisforced pitch/loudness

multi-channel output

Controlby MIDI notes

by envelopes, sequencer

by live sound analysis

39