Outstanding Questions

RITMO’s Objectives and Research Questions

Primary objective:

· Provide a comprehensive and interdisciplinary understanding of the perceptual and cognitive mechanisms underlying the human capacity to execute and experience rhythm.

Secondary objectives:

· Generate new insights into rhythmic structures and features of music, human bodily motion and audiovisual media.

· Establish a link between the basic structures and features of rhythmic phenomena in the world and within the (embodied) mind.

· Develop new theoretical frameworks for exploring the complexity of rhythmic processes in human life across disciplines.

Research problems:

Despite the importance of rhythm, there is still no comprehensive multidisciplinary theory spanning the wide range of rhythmic phenomena in human life. In RITMO we will focus on the cognitive mechanisms underlying the human capacity to experience and execute rhythm, investigating rhythmic processes within and between humans, as well as the results of such processes. We will ask the following questions:

1. What are the basic structures and features of rhythmic phenomena? 2. What are the links between these structures and features and the ways in which time

is organised in human perception and cognition? 3. What are the cognitive mechanisms underlying the experience and execution of

rhythm? 4. How can philosophical, psychological, and historical explorations of time be revisited

through new interdisciplinary perspectives on the experience of rhythm?

Proposals from the TIME cluster

Is a new theory of prediction and expectation processes needed to explain

musical pleasure?

Short background and why it is not answered:

Several have argued that emotional effects, including those related to aesthetic experiences,

crucially rely on the resolution of schematic expectancy violations or prediction errors (see,

for example, Koelsch, Vuust, and Friston 2019, Huron 2006, Meyer 1956, and Mandler

1984). From a music perspective, such theories are problematic for several reasons:

Expectancy violations in music that are not resolved are ubiquitous and have arguably

high aesthetic value.

While the current theories are operating with two successive phases––first expectancy

violations, then resolution––, music (i.e. rhythmic patterns) may operate with two

parallel expectancy schemes in which one is resolved while the other is violated.

We appreciate music’s atypical features even when, after repeated listing, it holds no

surprises (which either eliminates expectancy violation or resolution)––the so-called

Wittgenstein puzzle (Dowling & Harwood 1986, Huron 2006, Wittgenstein 1966).

Scholars have tried to answer the first question by, for example, arguing for an embodied fill-

in of missing parts (Vuust & Witek 2014 and Witek 2016), and the third question by, for

example, arguing that we differentiate between schematic lifetime-formed expectations and

veridical memory-based expectations (Bharucha 1994). Yet, these contributions require

supplementing, and problematization, by an interdisciplinary theory attentive to the broad

range of scholarship from cognition, embodied cognition, psychology, and neuroscience;

informatics and technology; and the arts and humanities; and that considers the full range and

complexity of relevant musical examples.

Why it is crucial:

A new theory of prediction and expectation processes is crucial to explain musical pleasure,

including the enduring fascination with cross-rhythms, polyrhythms, syncopation, and

atypical microtiming; the attraction of time signature alternation and non-isochronous meter;

and the obsession with fragmented rhythms, multitextured soundscapes, and sample-based

music.

References:

Bharucha, Jamshed, 1994: “Tonality and Expectation,” in Rita Aiello (ed), Musical Perception, pp. 213–39.

Oxford/New York: Oxford University Press.

Dowling, W. Jay and Dane L. Harwood, 1986. Music Cognition. Academic Press, INC.

Huron, David. 2006. Sweet Anticipation: Music and the Psychology of Expectation. MIT Press.

Koelsch, Stefan, Peter Vuust, and Karl Friston, 2019. “Predictive Processes and the Peculiar Case of Music.”

Trends in Cognitive Sciences, 23/1: 63–77.

Mandler, George, 1984. Mind and Body. New York: Norton.

Meyer, Leonard, 1956. Emotion and Meaning in Music. Chicago: University of Chicago Press.

Vuust, Peter and Maria A.G. Witek, 2014. “Rhythmic complexity and predictive coding: a novel approach to

modeling rhythm and meter perception in music”. Front. Psychol., 01.

Witek, Maria A.G. “Filling In: Syncopation, Pleasure and Distributed Embodiment in Groove”. Music Analysis,

36/1.

Wittgenstein, Ludwig, 1966. Lectures and Conversations on Aesthetics, Psychology and Religious Belief.

Berkley: University of California Pres.

Musical absorption in shared space and time Simon Høffding, Nanette Nielsen, Bruno Laeng, Jonna Vuoskoski

Background:

We know that engaging with music – whether performing or listening – has a plethora of

positive effects on the human being. To a large extent we do not know why. More

specifically, we need to understand the interactions between 1) Emotional arousal/affective

attunement 2) focused attention and reflection 3) a changed sense of agency.

Overall Questions:

What is musical absorption and what are its physiological correlates?

What is the physiological and phenomenological rhythmicity in shared musical absorption?

How might an understanding of musical absorption contribute to an understanding of time

consciousness?

Sub-question:

What is the relation between emotions, attention, and a changed sense of agency with regard

to musical absorption?

Outcomes:

- Clarifying physiological and behavioral components of absorption.

- Disambiguating the notion of attention, by constructing a hierarchy of different kinds.

- Evaluating what it means to "share musical time and space".

- Using ecologically valid studies of music that combine philosophical (phenomenological),

qualitative, and quantitative methods could potentially revolutionize consciousness studies.

- Developing an empirically-informed theoretical framework for interrogating the positive

effects musical absorption has on human beings

Methods:

Pupillometry and eye-tracking, heart rate and respiration measures, EEG, phenomenological

interviews.

Outstanding question – a sketch

(Erling E. Guldbrandsen, supported by Rolf Inge Godøy, Bruno Laeng and Anne Danielsen)

Main question:

In musical listening: How is the perceptual object organized and constantly re-organized in

the listener’s mind through longer stretches of musical time?

Sub-questions:

What is the role of expansion and contraction (Deleuze) and of tempo fluctuations in the

performed and perceived organization of musical time?

What is the role of memory and expectation (Husserl) in the ebb and flow of musical build-

ups through long-hauled processes of musical time?

What is the role of presence and absence (Derrida); that is: the role of the «now» as opposed

to what is no longer present, or not yet present, in the music?

What kind of method and language (scientific, phenomenological, interpretative, critical) (cf.

Wittgenstein; Adorno; dynamic theory) can be used to account for performances and

experiences of long-hauled musical time?

Why is it crucial:

Fairly often, music is appreciated through longer stretches of time (cf. works, concerts,

operas, films, musical sets, jam sessions, playing lists, albums). At the same time, certain

current theories of rhythm and time in music seem to focus mainly on shorter time stretches

and indeed, on micro-rhythmical differences. Whereas this is clearly a highly valuable

research, there is also a need to investigate the constitution of musical objects through longer

stretches of time. This may actualize the role of memory and expectation in human

understanding and in the experience of meaning, and may also bring up the age-old

ontological question of where a longer piece of music actually resides: To what extent is it a

purely intentional object, residing in a kind of mental representation, «hors-temps» as it were;

to what extent is it, conversely, a question of actual process in the here-and-now (cf. a

Bergsonian durée), and what is the deeper sense and function of memory and time in the

constitution of a musical process like this? Hence, a deepening research on perceived musical

process through time may also shed light on the understanding of human consciousness, self-

reflection, and the philosophy of time.

Proposals from the MOTION cluster

Outstanding questions: Sum-up of brainstorming What is rhythm? What does rhythm mean in different fields? Do we need meter to experience rhythm? Does a rhythm need to be experience? How/Why is rhythm important for humans? Is there a degree of rhythmicity? Can something be slightly rhythmic? How many pulses do you need to generate rhythm? Is periodicity a requirement for a rhythm? What level of periodicity? What about additive rhythms/polyrhythm? What are the metaphors of rhythm What is the ecological perspective of rhythm? What is similarity in music-related/ rhythmic movement? To what extent does the answer to this question depend on the type of movement (e.g. dance, walking, drumming, exercising)? How can we quantify these aspects? How are bodily rhythms related to the experience? What is the relationship between our experience of continuity and discontinuity? Physical world, embodied cognitive experience and machine representation? How does music mediate judgment of time durations? Why do we conceptualize time durations? What makes us tune in/synchronize in a new place/culture? (dislocation) Sense of feeling in balance? Related to rhythms of the body? Related to sound? How do the rhythms of a particular location influence our sense of being/wellness? How can we model rhythmic behaviour? What are we missing to be able to model rhythmic behaviour? How can we model the noise component of the world? What would still be missing? What are rhythmic (movement) features? How do we execute rhythms? Do they need to be conscious? How can we model the constraints of the human body when executing rhythms? How do different kinds of modalities and motion influence music production and perception? Do we need to start from a human perspective when developing machine systems? What are the creative capabilities of machines? How can computers learn to “read” body language/bodily expressions? Can computer agents learn this capability? How can we integrate computational models of rhythm in machines / robots? How can we model/learn intent and similarity in movement and sound? What is the role of rhythm in music performance and perception? What makes music multimodal? Does music really make us move? Why and how? Is there a multimodal mapping between sound and movement? How does rhythm influence other musical characteristics? Can we extrapolate a cognitive separation between speech and music from related movement?

Individual Contributions How can computers learn to “read” body language/bodily expressions? (Kristian) Bodily expressions influence strongly how a human observer perceives a verbal statement. We learn to know the body language of individuals, and of our own culture. When trying to understand the body language of strangers or new cultures, we may misunderstand initially, but gradually learn by exposure. Can computer agents learn this capability? (Similar questions: Can computers learn to understand sarcasm? or Musical irony?) What is similarity in music-related/ rhythmic movement? (Kristian) Humans have an intuition about degrees of similarity in movement, but what aspects of movement influence our notion of similarity? To what extent does the answer to this question depend on the type of movement (e.g. dance, walking, drumming, exercising)? How can we quantify these aspects? What makes music multimodal? (Rolf Inge) In musical experience, what is due to sound, to body motion, to vision, to sense of touch, or other senses? Or: is a pattern (melodic, rhythmic) in music just a pattern of sound events, or is it just as much a pattern of body motion? What is the relationship between our experience of continuity and discontinuity? (Rolf Inge) Music may be perceived as a continuous stream, yet we also tend to parse this stream into somehow meaningful events. What are rhythmic (movement) features? (Victor) Do robots need rhythm? (Kyrre) As rhythm is omnipresent in human life – both on a biological and a cultural level, we believe that rhythm is a phenomenon that is essential/advantageous for human behavior, and not a mere bi-product of other processes. The ability to perceive different types and levels of rhythm, adapt to them, to act in rhythm, and to predict future events, should contribute to more efficient communication and execution of tasks [EXAMPLES]. While robot technology has a vision of achieving human-level or better performance in many tasks, there has been little focus so far on rhythmic capabilities. There are obviously engineering efforts into precise timing of behavior and motion, but less in terms of a more holistic “sense” of rhythm: integrated rhythmic perception/adaptation/action/prediction processes, and if this could be beneficial for robots in the same way as humans benefit from rhythmic capabilities. How can we integrate computational models of rhythm in machines / robots? Does integrated rhythmic capabilities in robots improve performance in a) human – robot interaction, b) robot-robot interaction, c) robot-environment interaction? Does music really make us move? (Alexander) Why and how? How can we model/learn intent and similarity in movement and sound? (Benedikte) Is there a multimodal mapping between sound and movement? How do we execute rhythms? (Cagri) Conscious vs unconscious?

How can we model the constraints of the human body when executing rhythms? (Cagri) How does music mediate judgment of time durations? (Tejaswinee) Why do we conceptualize time durations? Are they continuous or discontinuous? How does rhythm influence other musical characteristics? Can we extrapolate a cognitive separation between speech and music from related movement? What makes us tune in/synchronize in a new place/culture? (Ximena) Sense of feeling in balance? Related to rhythms of the body? Related to sound? How do the rhythms of a particular location influence our sense of being/wellness? How are bodily rhythms related to the experience? What is the ecological perspective of rhythm? How do different kinds of modalities and motion influence music production and perception? (Qichao) Are we able to make a machine indistinguishable from a human? (Jim) The turing test, developed by Alan Turing in 1950, is a test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human. Turing proposed that a human evaluator would judge natural language conversations between a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in a conversation is a machine, and all participants would be separated from one another. The initial test was planned for conversation limited to a text-only channel. If the evaluator cannot reliably tell the machine from the human, the machine is said to have passed the test. Later there has been proposed more advanced versions of the test using video conference or even a real robot. We could apply the test for modalities in RITMO like motion or sound/rhythm. This is a crucial outstanding question since it would offer machines that can blend seamlessly with humans and offer effective human – machine interaction. Can robots design and make other robots? (Jim) Robots are today to a large extent manually designed and assembled. Artificial intelligence techniques allow for automating the design process, while 3D-printing technology combined with robot manipulators provide a solution for automated assembly. Still few demonstrations of a fully automated design and assembly process have been made. This is a crucial outstanding question since it would offer machines be self-designed and manufactured resulting in gradually more advanced machines that can solve more complex problems.

Proposal from RHYTHM/MOTION (Agata) What makes a person move to music? Why some people can’t help moving to music? To what extent can we control our physical responses to music? Why some people like to dance more than others?

Where is the border between pure auditory rhythms and musical rhythms? 1. How to decide whether what we hear is just a simple auditory pattern or a musical structure? 2. Do they affect our body movement differently? There are many studies that deal with this subject in context of rehabilitation (leading to no simple conclusion, Wittwer et al., 2013), but not in context of spontaneous bodily reactions, which are not related to any sensorimotor synchronization task. Also, typically only “synthetic metronome versus music” approach is applied, leaving out all options that lay in between. 3. Does a synthetic metronome sound have a different impact on a body movement than “organic” metronome sound, e.g. of a drum? If yes, in what ways and why?

Can animals make music? (or: is there non-human music?) T. Fitch proposed “four principles of bio-musicology” (2015) for a constructive scientific discussion on different aspects of animal behavior that can be classified as musical. The main question of the existence of “animal music”, however, remains unanswered (there are strong arguments in both directions). Other questions one can ask are: What conditions are necessary for an animal to be capable of, or to be motivated for, displaying signs of musicality? Are there sounds that have aesthetic qualities for given species, and if yes, do they make structures out of them? Does “possession” of musicality necessarily have an adaptive value (e.g. for the purpose of communication or mating)? Are there instances where animals make sounds purely for their own pleasure? Is obtaining pleasure an adaptive value in itself? And how can we know? Wittwer, J. E., Webster, K. E., and Hill, K. (2013). Music and metronome cues produce different effects on gait spatiotemporal measures but not gait variability in healthy older adults. Gait & posture 37, 219–222 Fitch, W. T. (2015). Four principles of bio-musicology. Philosophical Transactions of the Royal Society B: Biological Sciences, 370(1664), 20140091.

Proposals from the RHYTHM cluster (incl. comments from cluster members) A. Predictive processes in the brain (fRONT Neurolab)

The overarching outstanding question of our research activity is; “How does the brain implement the predictive processes that facilitate goal-directed cognition and behavior?”. Following is a list of more specific, but related, questions. These questions are examples of how our overarching question is relevant for the research at RITMO.

1) What is the hierarchical structure underlying predictive processes in the brain and is there a leading role of the frontal cortex? Comment: What about polymeter?

2) What is the role of rhythms in the brain and in the environment for prediction of the

“what”, “when” and “where” of human cognition? (also related to E)

Comment: What about why and how? Higher order aspects of prediction. The role of pleasure.

3) What is the role of brain oscillations in the communication between specialized

distant brain regions, forming a functional network?

4) What is the core computation (at the neuronal level and population/network level) supporting predictive processes in the human brain?

B. Meter, awareness & distributed attention (George)

Is awareness an obstacle to meter perception and rhythm production? Is distributed or focused attention more suitable for the emergence of meter and the sensation of beat? What role does attention play? What kind of attention? The relationship between awareness, attention and performance? Comment: Related to findings from attentional blink study (selective attention benefit from distributed attention/attending to the whole). (Merve) Comment: What is distributed attention? Does it exist – different name? Divided attention – auditory streams. Deutsch & Deutsch? The questions are intertwined. Distributed attention results in better estimation and prediction of durational patterns based on statistical properties. Such a mechanism is close to our understanding of meter perception: a predictive mechanism that is based on overall qualities of music and not on a single sound object, duration or auditory stream. At the same time, a distributed attention would lead to poor representations of the objects that make up the rhythm, or at least to representations not accessible to our consciousness. In

contrast, a focused or selective attention creates concrete representations of objects accessible to consciousness but perhaps is of little help in a complex (sound) environment such as music that encompasses several components, multiple auditory and rhythmic streams (with micro-rhythmical relations and polyrhythms), or even components belonging to various modalities, such the haptic and kinesthetic perception of body movement. Importantly, a focused attention mode would result in weak estimations of the statistical qualities of the overall rhythm and timing of events. How could a distributed attention mode lead to the emergence of meter? How aware are we of the properties of the numerous objects (and streams in different modalities) that lead to meter? How is meter that emerges from the selective focus on a single object or stream different from meter that results from a more distributed mode of attention? What is the role of attention in performance and rhythm production? And ultimately, what is the role of awareness and consciousness to the perception of meter and the sensation of beat? Distributed attention outside the visual modality or its implications for the timing of events have not been studied enough. The above questions and hypotheses need to be better understood in order to better understand the role of consciousness and awareness in musical experience. Polymeter (Anne) / simultaneous auditory streams and attention (Kjell-Andreas) How is polymeter (such as, for example, duple and triple pulse going on simultaneously) processed in the brain? How does attention at various levels affect the meter experience of polymeter? Alternative models:

a) Non-hierarchical processing / distributed attention?

b) Attention switches between the two pulses.

c) Hierarchical relationship?

The relationship between neural dynamics and actual experience. The role of body movement. Cultural factors of perception and cognition, learning, plasticity C. The temporal binding problem: What is now? (Bruno)

• What makes our minds (brains) see and hear and feel events as happening at the

same time, or before or after each other? • A counterfactual scenario: What if our brain used its molecules as ultra-precise

atomic clocks that track the passage of time to its infinitesimal bits and what if it could also time-tag precisely each of its signals: would this give us a better perception of physical reality? Would we be able to act more effectively? Would we be able to understand and produce even more extraordinary music?

The binding problem in the temporal domain. Prediction could be an explanation.

Relationship between different streams/integrating streams. Now has extension – not a point in time! D. Temporal precision in auditory processing (Anne): Not only the extension but also the

temporal resolution of the now varies Hypothesis: The temporal precision with which the perceptual apparatus processes auditory information is very flexible and varies with/adjusts to the given stimulus and the context (cf. the beat bin hypothesis). That is, temporal precision not better than the one needed for the actual task. As a consequence we can observe:

- Effect of priming on the detection of timing deviants - Habituation in the detection of timing deviants

Related to Maja’s experiment on statistical learning. Difference between perceiving change compared to perceiving stasis – trajectories, tempo changes? Slow and fast time? Relationship to motion? Perceptual time stretching. E. Feature integration versus feature segregation at the micro level of auditory

perception (Gui and Anne)

Are auditory features/dimensions such as, for example, timbre, pitch, timing, duration and intensity processed as part of a whole/auditory object at the basic level of perception or is each feature processed separately? If so at what point are they integrated? How do such auditory features/dimensions (eg. timing, pitch, intensity, duration) interact at the perceptual level? Can current models of interaction among different dimensions of auditory perception account for findings from various related performance/psychophysical/neurological studies, or are new models and experimental paradigms possible/needed? Related to the when what and where question? (What is the role of rhythms in the brain and in the environment for prediction of the “what”, “when” and “where” of human cognition?) F. About Musicality (Bruno)

• Why is music (listening, but also playing) a self-rewarding activity? • A counterfactual scenario: What if our brain suddenly lost the ability to enjoy music

or find it meaningful, so that the music and its instruments would become records of rituals past: Would our society and our minds be very different? Would our existences go on just as well (or bad) as before?

All kinds of arts?