Music Everywhere Project Half Presentation

PowerPoint Presentation

Halves Presentation Fall 2016AR-Assisted Piano Learning

AR-assisted piano learning using a MIDI-enabled piano with a MS HoloLens.

MusicianMusic Everywhere Team

Seth Glickman Byunghwan Lee Shantanu DasFu Yen HsiaoProgrammerProgrammerUI Designer

Faculty advisors.

Pitch Project InspirationTo explore how Augmented Reality technology can be used to assist in the learning of a musical instrument.

Surveys and Outreach

ReadingmusicRhythm ortimingTwo-hand coordination

Student ResponsesTeacher Responses37 Students11 TeachersWillingness to use technology in lessonsBarriers to successful piano learning27students volunteered to playtestHighest 3 responses

72.7%54.5%

Big PictureBest use of the AR tools we developed

AR Projection Interface

?QuartersPiano and HoloLens Connection

WebRTCWhat have we learned. Show the picture shows the problem

Quarter Semester Feedback

Top notes from faculty walkthroughs and sitdowns

What style(s) of music will you be working in? Is reading music required?Choose an area to investigate that augmented reality is well-suited to address.Experiment with animated hand as instructional device.Focus on precision in keyboard overlays.













Big Picture

Remote LearningBest use of the AR tools we developed

AR Projection Interface

?Artistic ExperienceScore ReadingImprovisationImprovisationScore ReadingQuartersPiano and HoloLens Connection

WebRTCWhat have we learned. Show the picture shows the problem

HoloLens and Music Improvisation

UI design informs users which notes will sound good, and change as the song progresses.Animated band provides music for users to play along with (and solos to imitate.

Research: Improvisation as Music InstructionHow do musicians who dont read music learn to play?The parallel paths of score readers and those who play by earLearning a to speak a musical language

Research on supplemental improvisation instruction included in music curriculum:

Can Improvisation be taught? A Call for free improvisation in our schools-Hickey, M.Northwestern UniversityUnveiling the Mysteries of Musical Spontaneity-Campbell, PMENC: National Association for Music Education

Developing Musicianship Through Improvisation-Azzara, CPh.D

Target Audience

Subject Matter Experts

Jack KurutzMavis PanAmaury MoralesPittsburgh Music AcademyBrooklyn College, Afro-Latin Jazz OrchestraCarnegie Mellon UniversityAdvisorReviewerReviewerTeaches basic improvisation with his private piano students, Jack is interested in having his 30+ students try an AR prototypeClassical and jazz instructor, performer and recording artist willing to be a sounding board for improvisation concepts

ConsultantWayne OquinThe Juilliard SchoolClassical piano instructor interested in contributing to improvisation model(s) for classical repertoireClassical composer and pianist will assist with high-level concepts and teaching strategiesMOST INVOLVEMENTLEAST INVOLVEMENT

Tina Blaine, Jack, Amaury, Mavis, Roger Powell, Dannenberg?

Related WorkPiano instruction

Music engagementAcademic AR piano studyLED-lighted keyboard hardware

Guitar instruction gameAR piano prototype

How different are you from these? AR + Real piano methodology. List each ones problem and goal. How you guys work differently in AR field?

Learning Objectives

Introduce skills of improvisation in various musical styles.Instruct improvisation techniques, phrasing and expression.Teach and reinforce two hand coordination.

Target Audience:Students 14+Intended as a supplement to standard piano lessons or to be used by a former piano student who wants to gain instruction and experience in improvisation.

4 styles

Development

Make lesson design process more flexible.

Development: Lesson Builder

We buildt lesson builder which can designer edit the lessone withouth coding.it provides flexibility and control of the structure for the lesson designerIt frees up programmers availablity to focus instead on building new features and new nodes.This is basic usage of lesson buidler


SibeliusExample: Score NodeImport, parse and draw musical scores in Unity.

One of the example as score node,Score node functions loading score data from composition program to unity.


Create node structure for all features.

Like score node, Each feature is a node inside of lesson builder.Right side of picture is one of our lesson.By using this, we achived more iterative and efficeint lesson design process

Development: Band System

Guitar PlayerBass Player

Drummer(Robert Johnson)(Ray Brown)(Albert Heath)

Blues Style Stage in 1950s

In order to enhance the improv experience, we design virtual band characters based on real musicians who will be playing with users. The band provide the sense of participation to users with responsive animations and representative musical theming. Take Blues band for an example, we got a cool guitar player, bass player and drummer in decent suits on a stage in a blues bar in 50s. Now, we are developing more music bands: Rock, Jazz and Classical in order to complete our music improv journey.

Development: Band SystemBass Player

(Kim Deal)Drummer

(Carter Beauford)Guitar Player

(Nigel Tufnel)Singer(David Lee Roth)

This is the rock style virtual band. More colorful, wild and complex.

Development: Band SystemPiano AccompanimentPiano ImprovisationPlaying With Band

It provides the experience which make user feel like playing with bandWe implemented central clock from which to sync all aspects of animation, sound and musical score in the project. Based on setted bpm or time scaleFor example , the drummer has sound clip and score data so it gonna hit drum right timing with musicThis can make more juicy experience to user.

Development: AR Piano Key Overlays

Place overlays to prevent players from looking down.Overlays should occupy minimal space in FOV to make space for the score and band.

We recognized very early in our development that an accurate and easily understandable overlays will be a very crucial for our project.Therefore we built a overlay interface where we could integrate new overlays very easily.UI feedback of the player - know when to play what to play - all filtered through AR - through a field of view when limited

UI: Overlay Iteration and Testing

Key overlay UI system plays a critical role in the improv experience and also an important design component in this project. How to guide users to hit the right keys with limited FOV efficiently without always looking down which is not the proper way to play piano and interact with the virtual band on the top of the piano. We hope them to look up

UI: Overlay Iteration and Testing

Key Overlay: Quarter / FullNo key overlay: Oval with line / TriangleLookup: Floating keys / Mirrored Keys

Last, we design floating keys hovering in the air and a mirrored keyboard in order to encourage users to look up.

22.2%25.9%

14.8%14.8%2.7%7.4%11.1%Playtest Results

53.8%38.5%

7.7%

Piano ExperienceOverlay PreferenceBeginnerIntermediate/Advanced

OtherTriangleMirrorFull KeyHalf KeyQuarter KeyFloating KeyOval with Lines

Out of the overlays we tested , the playtesters could identify the notes with high accuracy for all the overlay options. However, they preferred the mirror and the triangle overlay. Therefore from the results we can say that there were 2 favourites.

Development: Finger Assistance

Guest unfamiliar with the piano will often not play with proper fingers.

From our playtesting we observed, that Guest unfamiliar with the piano will often not play with proper fingers.

One of the benefits of using AR above other systems, is the ability to show hand projections for specific purposes.We can remedy the problem by showing a virtual hand that demonstrates finger technique and proper form

Development: AR Hand Display

Whats Next

Plan

ImprovisationImprovisationQuarters

Beyond the basic toolset

WebRTC

Stage 1 ImprovisationBand: BluesInstruction: Dom7 scale minor pentatonic scale Triads: I, IV, V Phrasing

WebRTC

Stage 1 ImprovisationBand: BluesInstruction: Dom7 scale minor pentatonic scale Triads: I, IV, V PhrasingStage 2 ImprovisationBand: RockReview: Dom7 scale minor pentatonic scale Triads: I, IV, V PhrasingAdd:Major scale Natural minor scale Triads: ii, iii, vi, vii Seventh: V7, vii7 Articulation Basic rhythms

Stage 3 ImprovisationBand: Jazz IntroductionReview:Major scale Natural minor scale Triads: ii, iii, vi, vii Sevenths: V7, vii7 Articulation Basic rhythmsAdd: ChromaticismModes Ear Training Musical Context Chart Reading Intermediate rhythms

Stage 4 ImprovisationEnsemble: ClassicalContinue: Ear Training Musical Context Chart Reading Intermediate rhythmsAdd: Sevenths: V of IV, V of V, Temporary tonicization Non-chord tones Arpeggios Chord function Tonic Dominant Sub-dominant Styles of classical improvisation

WebRTC

Project Scope

Level 5: Hands together

Level 4: Alternating hands

Level 3: Right hand improvisation

Level 2: Left hand accompaniment

Level 1: Improvisation basics

Showcase:Rock improvisation

Showcase:Jazz improvisation

BLUES

ROCK

JAZZ

CLASSICAL

Showcase:Classical improvisation

Technical Support

References, Feedback and Advice

Steve Audia, Jon Underwood, Bryan Maher, David Purta

Eric Brown, Anthony Daniels, Tina Blaine, Jessica Hammer, John Dessler, Scott Stevens, Jesse Schell, Michael Christel, Brenda Harger, Shirley Yee, Chris Klug, Tom Corbett, Heather Kelley, Richard Hilleman, Robyn Miller, Roger Powell, Jay Schneider, Jesse Stiles, Mavis Pan, Amaury Morales, Jack Kurutz, Daniel Cohen, Julian Korzeniosky, Drew DavidsonSpecial thanks to...

Early Prototyping

Music Everywhere Project Half Presentation

Engineering

Transcript of Music Everywhere Project Half Presentation