Robot Dance based on Online Automatic Rhythmic Perception Joo M. Lobato D. S. Oliveira In cooperation with: LIACC Laboratrio de Inteligncia Artificial e Cincia de Computadores INESC Porto Instituto de Engenharia de Sistemas e Computadores Coordinators: - Prof. Dr. Luis Paulo Reis - Prof. Dr. Fabien Gouyon Web Page: paginas.fe.up.pt/~ee03123 Mestrado Integrado em Engenharia Electrotcnica e de Computadores Index Introduction Objectives Tools Components System Architecture Experiments and Results Demonstration Conclusions Future Work 2 Autonomous Robotic Systems Robot Dancing Rhythmic-Interactive Systems MusicRhythm Cause - Effect Body Movement Autonomy Spontaneity Agent Environment Introduction SynchronismDynamismEntrainmentFlexibilityAdaptability 3 4 Introduction Considered Limitations Pre-programmed movements; Little flexibility; Poor adaptability; Inexistent or little human control; Direct applications of rhythmic perception models. 4 Introduction Objectives Develop a flexible dancing robotic (entertainment) system where a humanoid robot performs seemingly autonomous dance movements in synchrony to musical rhythm, without former knowledge of the music. 5 Synchronism Realism Dynamism Reactive Behavioural-Based Approach Effect Cause Rhythm Dance Tools 32-bits microprocessor; 8-bits 4MHz microcontroller; RAM and Flash memory; Monitor LCD 100x64; 16kHz Speakers; Connectivity - USB 2.0 and Bluetooth; Supports up to 3 servo-motors; Supports up to 4 ad-hoc sensors: colour, luminosity, sound, touch, ultrasonic, compass, between others. 6 Hardware Lego Mindstorms NXT Kit Tools Software Used Applications Marsyas (Music Analysis, Retrieval and Synthesis for Audio Signals) open-source framework constituted by various computational audio analysis blocks based on the most prominent published algorithms. MFC (Microsoft Foundation Class) to build the user interface; MATLAB numeric computation and programming environment. NXT Remote API C++ library to remotely control Lego NXT, via Bluetooth. 7 Tools Methodology Interconnection of Tools 8 Visual C++ (MFC) yy Marsyas Music Analysis Module Experimentation/Simulation MATLAB Engine MATLAB Engine Robot Control Module NXT Remote API Lego Mindstorms NXT Robot Dancing TCP/IP Sockets Bluetooth Human Control Module Windows OS GUI Components Dancing Humanoid Robot 9 RArm Motor LArm Motor Fan (spins with Head) Head Motor Body Motor NXT Bricks Colour Sensor Ultrasonic Sensor RLeg Motor LLeg Motor Components Dancing Environment 10 System Architecture System Framework Operation 11 Music Analysis Module Robot Control Module Human Control Module Rhythmic Events Audio Parameters Dance Composition Dance Movements D a n c e M o v e m e n t s Sensorial Events Sensorial Events Audio Signals WAV File TCP/IP Sockets System Architecture 12 Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication System Architecture 12 Multithreading Architecture Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Music Analysis Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client Music Analysis Module Multithreading Architecture Music Analysis Module 13 System Architecture Low-level rhythmic perception model based on onset detection. Onset Attack Decay Transient Music Analysis Module System Architecture 13 Low-level rhythmic perception model based on onset detection. Onset Attack Decay Transient Pre-ProcessingReduction Peak-Picking Onsets Location Detection Function Audio Signal (Pre-Processed) Original Signal Music Analysis Module System Architecture 13, Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client Music Analysis Module Multithreading Architecture Robot Control Module 15 System Architecture Dance movements embodiment based on a conjunction of rhythmic and sensorial events. Too Close OK Colour Event Red Green Yellow Blue Undef Yellow Red Green Blue Undefined Proximity Defined Dance Movement Rhythmic Event Medium Soft Strong Walk Waiting Events... STOP (Turn) 15 System Architecture Robot Control Module Thread 4: Dance Movements (Output Processing) Thread 4: Dance Movements (Output Processing) Thread 5: Brick 1 BT Communication Thread 5: Brick 1 BT Communication Thread 6: Brick 2 BT Communication Thread 6: Brick 2 BT Communication Thread 3: GUI + Asynchronous TCP/IP Server Thread 3: GUI + Asynchronous TCP/IP Server Robot Control Module Human Control Module Thread 1: Rhythmic Perception Model Thread 2: Synchronous TCP/IP Client Music Analysis Module Multithreading Architecture Human Control Module 16 System Architecture Robot Parameters Audio Input (14) Dance Movements List Dance Composition Control Panel Audio Parameters Event Visualization Connection Status (4) Velocity List 16 System Architecture Human Control Module Experiments and Results Rhythmic Perception Model Regulation Adaptive thresholding parameters analysis a) PN (Pitched Non-Percussion) excerpt using thres1 = 0.30; thres2 = 0.50; thres3 = b) PP (Pitched Percussion) excerpt using thres1 = 0.35; thres2 = 0.50; thres3 = d) CM (Complex Mixtures) excerpt using thres1 = 0.25; thres2 = 0.45; thres3 = c) NP (Non-Pitched Percussion) excerpt using thres1 = 0.30; thres2 = 0.40; thres3 = Experiments and Results Robot Dancing Performance Synchronism Realism Multi-threading architecture in a single processor. Lacks in the order of 1s Complexity of the output decision process, in the form of dance movements. Bluetooth communication overflow. Autonomous and reactive behaviour. The robot, through its dance motion, experiences the world directly, and this world directly influences its behaviour, through sensation Dynamism Great variety of possible dance style compositions, in a total of choreographies. Robots perambulation around the dance environment. System behaviour conducted by human decision. 18 Demonstrations 19 Conclusions Limitations: Synchronism limited by the Bluetooth communication system and due to the multithreading architecture in a single processor; Dynamism limited by the robots degrees of freedom, dependent of the number of supported motors (a maximum of 3 for each NXT Brick); Realism dependent from the former two factors: Reactivity dependent of synchronism Autonomy enforced by dynamism. 20 Flexibility Merits Interactivity Reactivity Autonomy Adaptability Easiness Future Work 22 High-Level Rhythmic Perception Models Spatial Rhythmic Perception Models Multi-Robot Cooperative Dancing Individual dance Low-Level Rhythmic Perception Models Flexible Interactive GUI Cross-Modulation Future Work 23 28