A Declarative Language for Dynamic Multimedia Interaction...

MultimediaInteraction in utcc

Olarte & Rueda.

A Declarative Language for DynamicMultimedia Interaction Systems

Carlos Olarte Camilo Rueda

LIX, Ecole Polytechnique Paris.Pontificia Universidad Javeriana Cali.

IRCAM.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

DynamicInteractive Scores

MusicImprovisationSystems

ConcludingRemarks

Motivation

I Multimedia interactive systems aims at devising waysfor the machine to be an active partner in a collectivebehavior constructed dynamically by many actors.

I The machine must always actively adapt its behavioraccording to the information derived from the activityof the other partners. For example:

1. Interactive Scores : defines a hierarchical structurewhere loosely coupled music processes may change theirproperties in reaction to stimulus from the environment.

2. Musical improvisation : A style-learning /improvisationsystem is a reactive system where several learning andimprovising agents react to information provided by theenvironment or by other agents.

I Challenges:

1. Complex coordination patterns of multiple agents.2. Ensuring the correctness of the system.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

I Challenges:

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

I Challenges:

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

I Challenges:

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

Concurrent Constraint Programming (CCP):

I A declarative model for concurrency tied to logic .

I It benefits from the large body of reasoning techniquesfrom both process calculi and logic.

I Universal timed CCP (utcc) extends CCP for mobilereactive systems :

1. Connection with logic is preserved (declarative ).2. The calculus is deterministic (elegant denotational

semantics, e.g. closure operators).3. A powerful synchronization mechanism based on

entailment of constraints (information available).

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

Concurrent Constraint Programming (CCP):

I A declarative model for concurrency tied to logic .

I It benefits from the large body of reasoning techniquesfrom both process calculi and logic.

I Universal timed CCP (utcc) extends CCP for mobilereactive systems :

1. Connection with logic is preserved (declarative ).2. The calculus is deterministic (elegant denotational

semantics, e.g. closure operators).3. A powerful synchronization mechanism based on

entailment of constraints (information available).

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

In this paper we argue for utcc as a declarative model fordynamic multimedia interaction systems:

1. Partial Information (constraints) allows for thespecification of weakly defined temporal relationsbetween agents/events.

2. Simple coordination and synchronization of agentsbased on constraints (blocking asks).

3. The expressivity of utcc allows for more flexible anddynamic systems.

4. Relying on the underlying temporal logic in utcc,non-trivial properties of the model can be verified .

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Motivation

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Contributions (Applications)

We present two applications:

1. A model for dynamic interactive scores whereinteractive points can be defined to adapt thehierarchical structure depending on the informationinferred from the environment.

2. A model of a music improvisation system based on theFactor Oracle that scales up to situations involvingseveral players, learners and improvisers.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Contributions (Applications)

We present two applications:

1. A model for dynamic interactive scores whereinteractive points can be defined to adapt thehierarchical structure depending on the informationinferred from the environment.

2. A model of a music improvisation system based on theFactor Oracle that scales up to situations involvingseveral players, learners and improvisers.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Outline

The utcc calculus

Dynamic Interactive Scores

Music Improvisation Systems

Concluding Remarks

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Outline

The utcc calculus

Concluding Remarks

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Concurrent Constraint Programming

I Von Neumann’s store is replaced by a store of partialinformation represented as Constraints (e.g. x < 10).

I The type of constraints is given by a Constraint System(e.g. first-order formulae over a given signature).

I Agents monotonically add (tell ) constraints. Theysynchronize via blocking asks querying the store.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

tell temperature > 42 ask temperature = 50 then P

ask 0<temperature<100 then Q

temperature=?

tell temperature < 70

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

ask temperature = 50 then P

42 <temperature

tell temperature < 70

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

42 <temperature<70

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

42 <temperature<70

Remains Blocked

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Constraint Systems.

I A constraint system is a tuple 〈∑,∆〉 where

∑is a

signature and ∆ a consistent first-order theory over∑

I Constraints are first-order formulae over∑

I Entailment relation c ` d holds iff c ⇒ d is valid on∆. c ≡ d iff c ` d and d ` c

I C denotes the set of constraints modulo ≡ in 〈∑,∆〉

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The tcc Model

1. Receives a stimulus (i.e a constraint) from theenvironment.

2. Computes a CCP process in the current time-unit andwait for stability.

3. Responds with the resulting store.

4. Executes the Residual process in the next time-unit.

* Note: Stores are not automatically transferred from atime unit to the next one.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The tcc Model

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The tcc Model

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The tcc Model

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The utcc calculus

Syntax

I tell(c) : adds c to the store in the current time interval.

I (abs ~x ; c) P : executes P[~t/~x ] for each ~t s.t. c[~t/~x ] canbe deduced from the current store.

I (local~x ; c) P : behaves like P but the informationabout variables in ~x is local to P

I next P : executes P in the next time unit.

I unless c next P : executes P in the next time unit if ccannot be entailed now.

I p(~x) : procedure call (derived operator).

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Operational Semantics

RT 〈tell(c), d〉 −→ 〈skip, d ∧ c〉RP

〈P, c〉 −→ 〈P′, d〉〈P ‖ Q, c〉 −→ 〈P′ ‖ Q, d〉

〈P, c ∧ (∃~xd)〉 −→ 〈P′, c ′ ∧ (∃~xd)〉〈(local~x ; c) P, d〉 −→ 〈(local~x ; c ′) P′, d ∧ ∃~xc′〉

RUd ` c

〈unless c next P, d〉 −→ 〈skip, d〉RR 〈! P, d〉 −→ 〈P ‖ next ! P, d〉

d ` c[~t/~x] |~t |= |~x |〈(abs ~x ; c) P, d〉 −→

˙P[~t/~x] ‖ (abs ~x ; c ∧ ~x 6= ~t ) P, d

γ1 −→ γ2

γ′1 −→ γ′2

if γ1 ≡ γ′1 and γ2 ≡ γ′2

〈P, c〉 −→∗ 〈Q, d〉 6−→

P(c,d)

====⇒ F (Q)

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

An Example

Let P be a process modeling a musician and Q animprovisation system :

Pdef= tell(play (A)) ‖ next (tell(play (G)) ‖ next tell(play (B))) . . .

Qdef= ! (abs x ; play(x)) R

When executing P ‖ Q, we observe, e.g., R[A/x ],R[G/x ],etc meaning that P and Q synchronized on the constraintplay(·): The note played by P (i.e. G ) is read by Q andthen processed by R.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Outline

The utcc calculus

Concluding Remarks

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Interactive Scores

Allen relations:

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Interactive Scores

C Score relations:A precedes B A meets DD overlaps B R contains CR contains D C contains A

I An Interactive Score is a pair composed of temporalobjects and structural and temporal (Allen) relations .

I Each object is comprised of a start-time, a duration,and a procedure (operational meaning).

I The idea: Dynamic changes in the hierarchy. E.g., if aninteraction does not occur, the composer may move theinterval to a similar musical context.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Dynamic Reconfiguration:

I Moving boxes.

I Adding/deleting intervals.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The Model

BoxOpdef= (abs id , d ; mkbox(id , d)) (local s) tell(box(id , d , s))

‖ (abs id ; destroy(id))(abs x , sup; in(x , id) ∧ in(id , sup))

unless play(id) next tell(in(x , sup))‖ (abs x , y ; before(x , y)) when ∃z (in(x , z) ∧ in(y , z)) do

unless play (y) next tell(bf(x , y))‖ (abs x , y ; into(x , y)) unless play (x) next tell(in(x , y))‖ (abs x , y ; out(x , y)) when in(x , y) do

unless play (x) next (abs z, in(y , z); tell(in(x , z)))

Constdef= (abs x , y ; in(x , y)) (abs dx , sx ; box(x , dx , sx ))

(abs dy , sy ; box(y , dy , sy ))tell(sy < sx ) ‖ tell(dx + sx ≤ dy + sy )

‖ (abs x , y ; bf(x , y)) (abs dx , sx ; box(x , dx , sx ))(abs dy , sy ; box(y , dy , sy )) tell(sx + dx ≤ sy )

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The Model

BoxOpdef= (abs id , d ; mkbox(id , d)) (local s) tell(box(id , d , s))

‖ (abs id ; destroy(id))(abs x , sup; in(x , id) ∧ in(id , sup))

unless play(id) next tell(in(x , sup))‖ (abs x , y ; before(x , y)) when ∃z (in(x , z) ∧ in(y , z)) do

unless play (y) next tell(bf(x , y))‖ (abs x , y ; into(x , y)) unless play (x) next tell(in(x , y))‖ (abs x , y ; out(x , y)) when in(x , y) do

unless play (x) next (abs z, in(y , z); tell(in(x , z)))

Constdef= (abs x , y ; in(x , y)) (abs dx , sx ; box(x , dx , sx ))

(abs dy , sy ; box(y , dy , sy ))tell(sy < sx ) ‖ tell(dx + sx ≤ dy + sy )

‖ (abs x , y ; bf(x , y)) (abs dx , sx ; box(x , dx , sx ))(abs dy , sy ; box(y , dy , sy )) tell(sx + dx ≤ sy )

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

The Model(2)

Clock(t, v)def= tell(t = v) ‖ nextClock(t, v + 1)

Play(x , t)def= when t ≥ 1 do tell(play(x)) ‖

unless t ≤ 1 nextPlay(x , t − 1)

Init(t)def= (wait x ; init(x)) do

(abs dx , sx ; box(x , dx , sx ))Clock(t, 0) ‖ tell(sx = t) ‖! (wait y , dy , sy ; box(y , dy , sy ) ∧ sy ≤ t) do Play(y , dy )

Systemdef= (local t) Init(t) ‖! Constraints ‖! BoxOp ‖ UsrBoxes

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

An Example

UsrBoxesdef=

tell(mkbox(a, 22)) ‖tell(mkbox(b, 12)) ‖tell(mkbox(c, 4)) ‖tell(mkbox(d , 5))tell(mkbox(e, 2)) ‖tell(into(b, a)) ‖tell(into(c, b)) ‖tell(into(d , b)) ‖tell(into(e, d)) ‖tell(before(c, d)) ‖whenever play(b) dounless signal next

tell(out(d , b)) ‖tell(mkbox(f , 2)) ‖tell(into(f , a)) ‖tell(before(b, f ) ‖tell(before(f , d)))

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Declarative Interpretation of utcc

I Processes defined by the user may lead to inconsistentstores: E.g. placing a box that exceeds the boundariesof the container.

I The idea: Using the declarative view of utcc processesas FLTL formulae to verify the model.

Definition (utcc logic characterization)

Let [[·]] be the following map

[[skip]] = true [[tell(c)]] = c[[P ‖ Q]] = [[P]] ∧ [[Q]] [[(abs ~y ; c) P]] = ∀~y(c ⇒ [[P]])[[(local~x ; c) P]] = ∃~x(c ∧ [[P]]) [[nextP]] = ◦[[P]][[unless c nextP]] = c ∨ ◦[[P]] [[! P]] = 2[[P]]

Theorem (Logic correspondence)

Let s = c1.c2.c3... ∈ ouput(P). [[P]] ` 3d iff there existsi ≥ 1 s.t. ci ` d

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Declarative Interpretation of utcc

I Processes defined by the user may lead to inconsistentstores: E.g. placing a box that exceeds the boundariesof the container.

I The idea: Using the declarative view of utcc processesas FLTL formulae to verify the model.

Definition (utcc logic characterization)

Let [[·]] be the following map

[[skip]] = true [[tell(c)]] = c[[P ‖ Q]] = [[P]] ∧ [[Q]] [[(abs ~y ; c) P]] = ∀~y(c ⇒ [[P]])[[(local~x ; c) P]] = ∃~x(c ∧ [[P]]) [[nextP]] = ◦[[P]][[unless c nextP]] = c ∨ ◦[[P]] [[! P]] = 2[[P]]

Theorem (Logic correspondence)

Let s = c1.c2.c3... ∈ ouput(P). [[P]] ` 3d iff there existsi ≥ 1 s.t. ci ` d

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Verification of the Model

We can verify, for example,

I [[P]] ` 3∃x ,dx ,sx ,y ,dy ,sy (box(x , dx , sx) ∧ box(y , dy , sy ) ∧in(x , y) ∧ sx + dx > sy + dy ): The end time of the boxy is less than the end time of the inner box x . I.e., thebox y cannot contain x .

I [[P]] ` ∀x(∃dx ,sx (box(x , dx , sx)⇒ 3 play(x)): All themusical structures are eventually played .

I [[P]] ` 3∀x(∃dx ,sx (box(x , dx , sx)⇒ play(x)): At somepoint all the boxes are playing simultaneously.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Outline

The utcc calculus

Concluding Remarks

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Learning a Style

I Learning a Style implies finding out what musicmaterial appears in what contexts .

I Capture music material during performanceMusic material: some sequence of music events.

I Identify contextsContext: repeated suffix of a sequence.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Factor Oracle

I A structure for efficient representation of all substrings(factors) of a string.

I Used to represent performed music sequences and alltheir contexts .

A FO is represented as a graph:

I nodes are referred to as states Si .I Arcs are partitioned in

I forward links (Si ,Sk), k > i , labelled with elements ofthe string.

I backward links, (Si ,Sj), j < i , that bear no label.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Factor Oracle: example

Figure: A FO automaton for s = aba

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Factor Oracle: properties

Let s = σ1, σ2, ..., σn

I States are S0, S1, ...,Sn

I All links (Si−1,Si ), 1 ≤ i < n, labelled with σi , arepresent.

I Some links (Si ,Sk), 0 ≤ i < k ≤ n, labelled with σk ,can be present.

I Some links (Si ,Sj), 0 ≤ j < i ≤ n, can be present(suffix).

For any factor p of s

I There is a unique path labelled p from S0 to some Si ,following factor links.

A suffix link (Si ,Sj) exists iff

I The longest repeated suffix of s[1..i ] is recognized in j .

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Let s = σ1, σ2, ..., σn

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Let s = σ1, σ2, ..., σn

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Factor Oracle: on-line construction

For each new input symbol σi (say σ4 = a):

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

A new state Si is added

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

A factor link (Si−1,Si ), labelled σi is created

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Starting from Si−1, suffix links are iteratively followedbackwards

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

until a state Sj where link (Sj , Sk) labelled σi exists... or there are no suffix links to follow (i.e. Sj = 0)

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

for each state Sm encountered in the iteration,create a factor (Sm, Si ), labelled σi

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

finally, create a suffix link (Si ,Sj)

Factor Oracle for s = abaa

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

A utcc model for the FO

I The data structure is represented as constraints of theform edge(x , y ,N).

I Each event (input) play(Note) triggers theconstruction of the new node and arcs.

I Synchronization is done semantically , through theavailable information in the store: It scales up tosituations involving several agents.

Step1(Note)def= tell(edge(i − 1, i ,Note)) ‖ Step2(Note, i − 1)

Step2(Note,E)def= when E = 0 do

(abs k; edge(E , k,Note))(tell(edge(i , k, ?)) ‖ next tell(ready))‖ unless ∃kedge(E ,K ,Note) next

(tell(ready) ‖ tell(edge(i , 0, ?))). . .

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Outline

The utcc calculus

Concluding Remarks

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Concluding Remarks

We argued for utcc as a declarative framework for modelingand verifying dynamic multimedia interaction systems.

I Synchronization mechanism based on entailment ofconstraints leads to simpler models that scale up whenmore agents are added.

I Systems can be formally verified by using theunderlying temporal logic in utcc.

Two models of non trivial interacting systems.

I Interactive scores where the composer can dynamicallychange the structure of the score according to theinformation derived from the environment.

I A simpler model of the FO that scales up when multipleagents are considered.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Concluding Remarks

We argued for utcc as a declarative framework for modelingand verifying dynamic multimedia interaction systems.

I Synchronization mechanism based on entailment ofconstraints leads to simpler models that scale up whenmore agents are added.

I Systems can be formally verified by using theunderlying temporal logic in utcc.

Two models of non trivial interacting systems.

I Interactive scores where the composer can dynamicallychange the structure of the score according to theinformation derived from the environment.

I A simpler model of the FO that scales up when multipleagents are considered.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Future work

I Guarantee reliable responses in time: Assessing thebehavior of utcc processes in real-time contexts.

I Provide a more principled notion of time where theduration of each time-unit can be related to the amountof computation involved in it.

I Enrich the FO model with probabilistic traversals of thegraph.

Olarte & Rueda.

Motivation

Outline

The utcc calculus

ConcludingRemarks

Thank you!

A Declarative Language for Dynamic Multimedia Interaction...

Documents

Transcript of A Declarative Language for Dynamic Multimedia Interaction...