Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

32
29/09/2011 1 Rhythmic Blueprints Design and Evaluation of Rhythmic Interaction Cumhur Erkut and Antti Jylhä Aalto University, School of Electrical Engineering Department of Signal Processing and Acoustics Tampere, Finland MindTrek 2011 29/09/2011 Overview Getting know each other Sensitizing: interactive rhythms in various scales Musical rhythms Social rhythms Elements of rhythm Pulse, beat, meter, and tempo Resonance, synchronization, entrainment Design and evaluation models, and how we use them Conclusions: Directions and Guidelines Remember: http://blogs.aalto.fi/rhythmicity/ 29/09/2011 2

description

Despite its central role in interactive media, guidelines for designing and evaluating rhythmic interactions are hard to find. The primary objective of our tutorial is that the participants will learn the fundamentals of the design and evaluation of rhythmicity in ambient and ubiquitous multimodal mobile interaction.The tutorial is organized by Cumhur Erkut and Antti Jylhä from the Aalto University. The target audience is the interaction specialists and software developers, who want to enhance their interactive applications with rhythmicity. A particularly suitable group is the mobile application developers. Students and general audience with background in HCI, interaction design, or audio signal processing were also welcome to attend. About 10 HCI, game, and interaction design researchers have followed the tutorial and created a lively discussion.

Transcript of Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

Page 1: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

1  

Rhythmic Blueprints

Design and Evaluation of Rhythmic Interaction

Cumhur Erkut and Antti Jylhä

Aalto University, School of Electrical Engineering

Department of Signal Processing and Acoustics

Tampere, Finland MindTrek 2011 29/09/2011

Overview

•  Getting know each other •  Sensitizing: interactive rhythms in various scales

–  Musical rhythms –  Social rhythms

•  Elements of rhythm –  Pulse, beat, meter, and tempo –  Resonance, synchronization, entrainment

•  Design and evaluation models, and how we use them •  Conclusions: Directions and Guidelines •  Remember: http://blogs.aalto.fi/rhythmicity/

29/09/2011 2

Page 2: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

2  

Michael H. Thaut, in Rhythm, Music, and the Brain, pp. 16-17

If we return briefly to the importance of temporal regulation for all our higher cognitive and motor functions, we may have very good reason to believe that rhythm in music, the element of temporal order, has a unique and profound influence on our perceptual processes related to cognition, affect, and motor function. Rhythm may enhance our brain operations through providing structure and anticipation in time. Rhythm may be one of the central processors to optimize our gestalt formation in the basic process of learning and perception.

29/09/2011 3

Rhythmic interaction

•  Time is of the essence •  A natural human capability

–  Walking, hammering, talking… –  Anticipation, mutual coordination

•  Examples –  Personal Orchestra {Borchers:2004hx} –  Virtual interactive humanoids {Nijholt:2008ty} –  Percussion robot Haile {Weinberg:2006wl} –  B-keeper {Robertson:2007tf} –  Hand clap interface {Jylha:2009gq}

29/09/2011 4

Page 3: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

3  

Demo: Sonically Augmented Table and Rhythmic Interaction {Pesonen:2010ur}

29/09/2011 5

Elements of Musical Rhythms

•  Pulses: events in a “pulse train” with regular temporal spacing –  The inter-pulse interval always the same –  The basis of rhythm perception {Thaut:2005te}.

•  Beats: audible pulse markings –  Sequenced events –  May deviate from exact pulse timings in slight shifts

29/09/2011 6

Page 4: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

4  

Elements of Musical Rhythms

•  Tempo –  Metric of (musical) rhythmic “speed” (rate of pulses/beats) –  Inversely proportional to inter-pulse interval –  Often measured as beats per minute (BPM) –  In music, never completely stable

•  Meter –  Defines the rhythmic structure in music (and vice versa) –  Often expressed as beats per measure

•  E.g., 3/4, 4/4 –  Relates to accentuation

29/09/2011 7

Related concepts

•  Vibration: Mechanical response of a body to an external stimuli

•  Frequency: Vibrations per second •  Resonance: Tendency to vibrate at a certain frequency •  Entrainment: A process between multiple bodies to align their

rhythmic resonances •  Synchronization: The quasi-stable state of entrainment

29/09/2011 9

Page 5: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

5  

Time-sensitive crowded scenes in movies

29/09/2011 10

Laws are simple: use in media?

29/09/2011 11

Page 6: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

6  

Sound synthesis, control, and hierarchical events

29/09/2011 12

ClaPD

29/09/2011 13

Page 7: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

7  

29/09/2011 14

Tutorial in a Nutshell …

29/09/2011 15

Page 8: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

8  

Hand clapping interface for sonic interactions (AM ’08 / CHI ’09)

29/09/2011 16

Applications

•  Potential in –  Games and entertainment –  Sound design –  New HCI schemes

•  Three example cases 1.  Hand-clap driven sampler 2.  Controlling music tempo 3.  Synchronizing a virtual audience

29/09/2011 17

Page 9: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

9  

Hand clap detection

Tempo estimation

Clap type identification

Application User Sound input

Sonic feedback

System dataflow

29/09/2011 18

Music tempo control

•  The user claps to control the tempo of music –  BPM of the user’s clapping is

mapped to that of the music

29/09/2011 19

Page 10: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

10  

Virtual audience

•  The user claps to synchronize a virtual audience with her clapping

•  The user can sync the audience with or without reference music

•  Interaction is immediate –  The user is part of the

clapping crowd

29/09/2011 20

Demo video

•  http://www.youtube.com/watch?v=7HLYGkayAGA

29/09/2011 21

Page 11: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

11  

Interface implementation with PD

•  Clap detection: [bonk~] (Puckette98)!

•  Tempo estimation: [rhythm_estimator] (Seppänen01)!

•  Combination: [clap_tracker]!

29/09/2011 22

Evaluation

•  Informal evaluation –  2 subjects tested the example applications –  Interface was found easy to use –  Both subjects found that the tempo of the virtual audience or the

music drove their clapping

•  Some latency appears in the system –  Mostly from buffering and computations –  Not reported as disturbing by the subjects

29/09/2011 23

Page 12: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

12  

Negotiation

•  Perceived tempo affected subjects’ clapping –  The user negotiates with the computer to set the tempo

•  Mutual coordination, ”dual-drive” –  Process analogue: musical ensemble

29/09/2011 24

Explorations of rhythmic interaction with dancers

{Erkut:2009wu} 29/09/2011 25

Page 13: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

13  

Design Blueprints: our version

Based on Z. Obrenovic, J. Abascal, and D. Starcevic, “Universal accessibility as a multimodal design issue,” Communications of the ACM, vol. 50, no. 5, pp. 83–88, 2007.

\cite{Erkut:2011ta} 29/09/2011 28

Rhythmic musical interface

29/09/2011 29

Page 14: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

14  

29/09/2011 30

iPalmas (AM ‘09)

•  An interactive Flamenco rhythm tutor for hand clapping

\cite{Jylha:2009uc} 29/09/2011 31

Page 15: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

15  

Flamenco

29/09/2011 32

Rhythm in Flamenco

•  Compas = meter in Flamenco •  Usually 12-beat cycles

–  Accentuation on certain beats depending on style, e.g. bulerias: X - - X - - X - X - X -

•  Lots of percussion –  Footwork, hand clapping, instruments…

•  Focal performer leads –  Improvisation requires communication

29/09/2011 33

Page 16: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

16  

Palmas

•  Supporting the compas

•  Palmero/-a •  Two types

–  Hard and soft •  Follow the focal

performer

http://www.youtube.com/watch?v=Yo38h7Wdc88

29/09/2011 34

Palmas skill requirements

1.  Sordas and fuertes 2.  Steady and accurate basic accompaniment 3.  Decorative clapping 4.  Starting to clap after silence 5.  Reacting to tempo changes 6.  Reacting to rhythmical cues

29/09/2011 35

Page 17: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

17  

System (1)

•  Flamenco palmas synthesis and tutor –  Teaches the skills –  Provides synthetic examples and accompaniment

•  User’s clapping as input •  Auditory and visual feedback

29/09/2011 36

System (2)

User Clap analysis

GUI

Clap synthesis

Visual feedback

Virtual tutor

29/09/2011 37

Page 18: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

18  

System (3)

29/09/2011 38

Synthetic palmas

•  ClaPD hand clap synthesis engine (Peltola et al. 2007) –  Enveloped noise burst –  Band-pass filter defines clap type –  Coupled oscillator model for rhythmic interaction

•  Pre-defined palmas patterns set synthesis parameters •  User and/or tutor set the tempo

29/09/2011 39

Page 19: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

19  

Analysis of clapping

•  Clap type –  Classification by template matching ( [bonk~] )

•  Accent detection –  Loudness-based

•  Tempo –  Probabilistic estimate based on inter-onset intervals

•  Tempo steadiness –  Temporal variance

•  Correct accentuation –  Comparison of user’s and tutor’s accent patterns

29/09/2011 40

Feedback (1)

•  Clapping sounds from the system –  Synchrony, accents, tempo changes

•  Numeric feedback and sliders –  Performance metrics

•  Visualization –  Dancing circles –  Transcription of the pattern

29/09/2011 41

Page 20: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

20  

Feedback (2)

29/09/2011 42

Prototype

•  Built on Pure Data (Pd) •  Synthesis of virtual palmeros •  Clap type training •  Basic compas training

–  Tutor speeds up when the user gets better -> Tempo changes trained as well

29/09/2011 43

Page 21: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

21  

Interlude: evaluation

Human System

29/09/2011 44

Interlude: evaluation

Human System Interaction

29/09/2011 45

Page 22: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

22  

What can/should be evaluated

•  Human factors –  Rhythmic capabilities

•  Perception •  Production

•  System factors –  I/O latency –  Computational complexity

•  Interaction –  Fluency, naturalness –  Etc.

•  The whole triptych

29/09/2011 46

Methods of evaluation

•  Qualitative –  Characterization of the capabilities/properties/phenomena

•  Understanding –  Interviews, observations, etc.

•  Quantitative –  Measurable quantities

•  For quantifying the qualitative attributes •  Also can be used to derive qualitative results

29/09/2011 47

Page 23: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

23  

Human factors

•  Sensorimotor synchronization (SMS {Repp:2005tb}) –  Synchronization between sensory stimulus and motor response –  Often metronome-based evaluation or simple rhythmic tasks

•  “tapping to the beat” –  Also, movement-to-music evaluation

•  How movement trajectories correlate with the rhythmic elements

•  Rhythmic capabilities –  Production: e.g., rhythmic stability, accentuation –  Perception: modal acuity –  Some people can be actually “rhythm-deaf” (Phillips-Silver et al. 2011)

29/09/2011 48

System factors

•  I/O latency –  Plays a role in rhythmic applications

•  In principle a constraint, but can be overcome in cyclic interactions

•  I/O modalities –  Auditory, visual, haptic –  Streaming (continous) vs. event-based (discrete)

•  Computational properties –  Complexity, required processing power, memory, …

29/09/2011 49

Page 24: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

24  

Evaluating interaction

•  Qualitative studies •  User tests

–  Often based on simple tasks

•  In-performance evaluation •  Hybrid methods

29/09/2011 50

However…

•  … evaluation of individual components is valuable, but does not necessarily explain everything Wholesome evaluation needed

29/09/2011 51

Page 25: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

25  

Case example: iPalmas evaluation (CMJ ’11)

•  Subjective experiment AND objective description –  Both qualitative and quantitative measures

\cite{Jylha2011:CMJ} 29/09/2011 52

iPalmas revisited

Audiorecording

Detection andanalysis

Audiofeedback

Visualfeedback

Metrics:- Tempo- Accent- Deviation- Time stamp

Text file log

Post-analysisTempo

Transcription,circles,numeric feedback

Hand claps

Synthetic hand claps,

User

Tutor

Audio Mostly 2011 Jylhä & Erkut 29/09/2011 53

Page 26: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

26  

Subjective experiment

•  To evaluate 1.  Human factors 2.  System 3.  Interaction

•  Rhythmic tasks with realistic rhythmic patterns –  Training, testing

•  Logging of measurable quantities •  Observations throughout the experiment •  Interviews, verbal comments •  Post-experiment questionnaire

29/09/2011 54

Evaluation of the first iPalmas system

•  Performed to evaluate rhythmic interaction and the system (Jylhä et al. 2011)

•  16 subjects •  4 patterns •  4 tutors (audio and audiovisual, adaptive and fixed tempo) •  Training phase and test phase, recall phase

Audio Mostly 2011 Jylhä & Erkut 29/09/2011 55

Page 27: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

27  

Evaluation of the first iPalmas system

Audio Mostly 2011 Jylhä & Erkut 29/09/2011 56

Evaluation of the first iPalmas system

•  Auditory information (clapping) the key to learning and performing –  The reverb disturbing to some

•  Transcription helpful •  Circles pretty but not very helpful •  Numeric FB useful only to some

–  Helped in “tuning in” to clapping the accents

•  Temporal variation in clapping –  Tutor stops subject speeds up

•  Adaptive mode helps performance

Audio Mostly 2011 Jylhä & Erkut 29/09/2011 57

Page 28: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

28  

Tempo speed-up and fluctuation

0 20 40 60

160

180

200

Time (sec)

BPM

Audio only tutor (steady)

Subject11Subject1

0 20 40 60

160

180

200

Time (sec)

BPM

Audio only tutor (adaptive)

0 20 40 60

160

180

200

Time (sec)

BPM

Audiovisual tutor (steady)

0 20 40 60

160

180

200

Time (sec)

BPM

Audiovisual tutor (adaptive)

\cite{Jylha2011:CMJ} 29/09/2011

58

Quantitative findings

•  Average time to start clapping: 10 cycles (40 s) •  Accentuation correctness

–  Audio-only: 68.1 % –  Audiovisual: 73.7 % –  Fixed tempo: 67.6 % –  Adaptive tempo: 74.2 %

•  Indication: for accentuated beats, IOI slightly longer (344.7 ms) than for non-accentuated beats (343 ms) –  More prominent with the adaptive tutor

\cite{Jylha2011:CMJ} 29/09/2011 59

Page 29: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

29  

Evaluation Blueprints: Summarize Qualitative, quantitative, and metrics

\cite{Erkut:2011ta} 29/09/2011 60

Direction1: SID

http://en.wikipedia.org/wiki/Sonic_interaction_design 29/09/2011 61

Page 30: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

30  

Direction 2: Interaction Gestalts and Attributes

{Lim:2009tw}

Interaction gestalt

User experience •  User experience

qualities

Interactive artifact •  Artifact

properties

29/09/2011 62

Directions 1+2 Combined: New basic sonic interaction design

http://www.room50.org/stefanodellemonache See, \cite{Rocchesso:2009wi}, and \cite{Franinovic:2009wl} 29/09/2011

63

Page 31: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

31  

Other directions

•  Social rhythms: H. Lefebvre, “Rhythmanalysis: Space, Time and Everyday Life,” Book, pp. 1–129, 2004 => CSCW => Social games

•  Rhythms and emotions: Thaut’05, but also recent CHI papers, e.g. \cite{Epp:2011hz}

•  Gamification! •  What else?

29/09/2011 64

References

•  C. Epp, M. Lippold, and R. L. Mandryk, “Identifying emotional states using keystroke dynamics,” in CHI’11, Vancouver, BC, Canada, 2011, pp. 715–724.

•  C. Erkut, A. Jylhä, and R. Discioglu, “A structured design and evaluation model with application to rhythmic interaction displays,” in Proc. New Interfaces for Musical Expression (NIME), Oslo, Norway, 2011, pp. 477–480.

•  C. Erkut, A. Jylhä, and I. Ekman, “Recent advances in exploring self-induced sonic interactions in the context of performing arts,” in Intl. Workshop on Haptic and Audio Interaction Design, 2009, pp. 1–2.

•  K. Franinovic, “Toward Basic Interaction Design,” available online at http://tdd.elisava.net/coleccion/25/franinovic-en

•  A. Jylhä and C. Erkut, “A hand clap interface for sonic interaction with the computer,” CHI-EA, Apr. 2009.

•  A. Jylhä, C. Erkut, I. Ekman, and K. Tahiroglu, “iPalmas - An interactive flamenco rhythm machine,” Proc. Audio Mostly, pp. 1–2, May. 2009.

•  A. Jylhä, I. Ekman, C. Erkut, and K. Tahiroglu, “Design and Evaluation of Rhythmic Interaction with an Interactive Tutoring System,” Computer Music Journal, vol. 35, no. 2, pp. 36–48. 2011.

29/09/2011 68

Page 32: Rhythmic Blueprints: A tutorial on Design and Evaluation of Rhythmic Interaction

29/09/2011  

32  

References

•  Y.-K. Lim, E. Stolterman, H. Jung, and J. Donaldson, “Interaction gestalt and the design of aesthetic interactions,” Proc.. Conf. Designing Pleasurable Products and Interfaces, pp. 239–254, 2007.

•  M. Pesonen, Sonically Augmented Table and Rhythmic Interaction, Master’s thesis, Aalto Univerity, School of Electrical Engineering, 2010.

•  B. Repp, “Sensorimotor synchronization: A review of the tapping literature,” Psychonomic Bulletin and Review, vol. 12, no. 6, pp. 969–992, 2005.

•  A. Robertson and M. Plumbley, “B-Keeper: A Beat-Tracker for Live Performance,” Proc. NIME, pp. 234–237, 2007.

•  D. Rocchesso, P. Polotti, and S. D. Delle Monache, “Designing Continuous Sonic Interaction,” Intl. J. Design, vol. 3, no. 3, pp. 13–25, May. 2009.

•  M. Thaut, Rhythm, Music, and the Brain. New York, NY, USA: Routledge, 2005.

•  G. Weinberg and S. Driscoll, “Robot-human interaction with an anthropomorphic percussionist,” CHI '06, Apr. 2006.

29/09/2011 69

Discussion

•  (Edward T) Hall's studies (on cultural rhythms) and (Saul) Greenberg's work on social rhythms (awareness)

•  Should the tutorial be called "interaction with rhythms" instead of rhythmic interaction?

•  Synesthesia as a resource? •  Behavioral change: negotiation

already used in game design (transfer of adaptation)

•  Adaptive coaching •  Language learning and

prononciation

29/09/2011 70