Feel the Music€¦ · feel the music 1experiment a formal experiment was conducted to prototype 2...
1
E XPLORING TACTILE MAPPINGS OF AUDITORY STIMULI AS A STEP TOWARD MAKING MUSIC MORE ACCESSIBLE FOR THOSE THAT ARE DEAF OR HEARING IMPAIRED . Feel the Music PROTOTYPE 2 EXPERIMENT 1 A FORMAL EXPERIMENT WAS CONDUCTED TO INVESTIGATE OUR EMOTIONAL RESPONSES TO TACTILE STIMULI AND IDENTIFY WHICH FEATURES OF HAPTIC SENSATIONS AFFECT THE USER ' S RESPONSE . RESULTS STIMULI • 42 S TIMULI WERE SINE AND SQUARE WAVES WITH VARYING FREQUENCY AND AMPLITUDE . • 60 STIMULI WERE GENERATED BY MULTIPLYING A 64H Z SINE WAVE WITH AN ENVELOPE WAVEFORM DEFINED BY A PA - RAMETER α. • 6 REPEATED STIMULI . • 20 PARTICIPANTS TOOK PART . • 108 TACTILE STIMULI WERE PRESENTED . • S TIMULI WERE HAPTIC SENSATIONS GENERATED BY A TACTOR PLACED ON EACH PARTICIPANT ’ S FINGERTIP . P ARTICIPANTS RATED THE STIMULI BY A ROUSAL ( INTENSITY ) AND V ALENCE ( PLEASANTNESS ). T HESE MEASURES ARE BASED ON THE C IRCUMPLEX M ODEL OF A FFECT , A WIDELY USED MODEL IN PSYCHOLOGY RESEARCH TO MEASURE EMOTIONAL RESPONSES . E XAMPLE STIMULI P ROGRESSION OF PROTOTYPES T ACTORS α VALUES FOR ENVELOPE STIMULI RESULTS T HE RESULTS LED TO THE CREATION OF A WEARABLE PROTOTYPE THAT EMPLOYED TACTILE SENSATIONS AS A MUSICAL INTERFACE . M APPINGS FROM MUSIC SAMPLES ONTO THE DEVICE WERE DESIGNED AND TESTED IN A SECOND EXPERIMENT . BY : ALICE HAYNES SUPERVISORS : DR J. LAWRY, DR J. ROSSITER E XAMPLE T ACTILE M APPING FOR “A SHES TO A SHES ’ A FTER MULTIPLE ITERATIONS AND MODIFICATIONS THE FINAL PROTO - TYPE CREATED WAS A SLEEVE CONTAINING : • 8 VIBRATION MOTORS WITH SMOOTH CASINGS PLACED IN SEQUENCE ALONG THE FOREARM . • A SQUEEZER MODULE FORMED OF AN ENCLOSED POCKET FILLED WITH AIR VIA A VACUUM PUMP ON THE UPPER ARM . T HESE WERE CONTROLLED BY AN A RDUINO L EONARDO WITH 3 STACKED MOTOR SHIELDS AND 1 SD CARD SHIELD ONTO WHICH MAP - PINGS WERE SAVED . T ACTILE MAPPINGS WERE PLAYED IN SYNC WITH MUSIC BY SENDING A START COMMAND VIA SERIAL FROM MATLAB AND PLAYING THE MUSIC SIMULTANEOUSLY . MAPPINGS C HOSEN MUSIC SAMPLES WERE CONVERTED INTO MIDI FILES FROM WHICH THE KEY MELODIES AND RHYTHMS WERE IDENTIFIED AND TRANSLATED INTO AN ARRAY OF INTENSITIES TO CONTROL THE MOTORS AND SQUEEZER OVER TIME . 7 MUSIC SAMPLES WERE USED , INCLUDING THE J AWS THEMTUNE , ‘I S THIS LOVE ’ B OB M ARLEY AND ‘A SHES TO ASHES ’ D AVID B OWIE . TESTING • 20 PARTICIPANTS TOOK PART . • 28 STIMULI WERE PRESENTED . • S TIMULI INCLUDED MUSIC SAMPLES , PURE TACTILE SENSATIONS AND SYNCRONISED COMBINATIONS OF THE TWO . P ARTICIPANTS RATED STIMULI BY V ALENCE ( PLEASANTNESS ) AND S E - LECTED ADJECTIVES FROM A LIST TAKEN FROM H ENVER ’ S A DJECTIVE C IRCLE TO DESCRIBE THE SENSATIONS . • M EAN V ALENCE RESPONSES ACROSS ALL PARTICIPANTS WERE STA - TISTICALLY THE SAME FOR M USIC ONLY STIMULI AND C OMBINATION STIMULI . (F) • M EAN V ALENCE RESPONSES TO T ACTILE ONLY STIMULI WERE STA - TISTICALLY LOWER THAN FOR THE STIMULI INCLUDING MUSIC . • P ER INDIVIDUAL SONG SAMPLE PREFERENCE FOR M USIC , T ACTILE OR C OMBINRTION PRESENTATIONS VARIED . (E) • O VER 60% OF RESPONSES PER SONG SAMPLE WERE STATISTICALLY SIMILAR FOR M USIC ONLY AND T ACTILE ONLY STIMULI SUGGESTING THAT MAPPINGS WERE EFFECTIVE . • A GE OF PARTICIPANTS DID NOT AFFECT THEIR RESPONSE TO MUSIC , BUT OLDER PARTICIPANTS GAVE STATISTICALLY LOWER V ALENCE RANK - INGS TO T ACTILE STIMULI THAN YOUNGER PARTICIPANTS . T HIS MAY BE DUE TO CHANGE IN SKIN SENSITIVITY WITH AGE . • A DJECTIVE RESPONSES GENERALLY MATCHED FOR M USIC AND T AC - TILE MAPPINGS BUT ALSO FOR PURPOSEFULLY NON - MATCHED STIMULI . I N CONCLUSION , THE PROTOTYPE SUCCESSFULLY DEMON - STRATED THE POTENTIAL FOR TACTILE STIMULATION TO BE USED AS A SENSORY SUBSTITUTE FOR ACOUSTIC MUSIC . • T HE EXPERIMENT FOUND THAT THERE IS A GENERAL NEGA - TIVE TREND BETWEEN A ROUSAL AND V ALENCE RESPONSES AS DEMONSTRATED BY THE MEAN RESPONSES ACROSS STIMULI . (A) • F REQUENCY HAS A SIGNIFICANT POSITIVE CORRELATION WITH ELICITED A ROUSAL RESPONSES . (B) • F REQUENCY IS SIGNIFICANTLY CORRELATED WITH 80% OF PARTICIPANT V ALENCE RESPONSES . H OWEVER , 6 WERE POSI - TIVE AND 10 WERE NEGATIVE . (B) • A MPLITUDE IS NOT SIGNIFICANTLY CORRELATED WITH A ROUSAL OR V ALENCE ALTHOUGH THERE IS A GENERAL POSI - TIVE CORRELATION WITH A ROUSAL AND NEGATIVE CORRELA - TION WITH V ALENCE . (D) • S QUARE WAVE STIMULI MORE AROUSING THAN S INE WAVE STIMULI WHICH MAY BE DUE TO THE POSITIVE CORRELATION BE - TWEEN ENERGY IN THE STIMULUS SIGNAL AND ELICITED A ROUSAL RESPONSE . • E NERGY IN A STIMULUS SIGNAL ALSO NEGATIVELY CORRE - LATED WITH V ALENCE RESPONSES . • R EPEATED STIMULI HAD A MEAN SPREAD OF RESPONSES ACROSS 15.5% OF THE SPACE PER PARTICIPANT . T HIS INDI - CATED RELIABILITY IN PARTICIPANT RESPONSES SINCE TESTS RUN FOR MULTIPLE RANDOM RESPONSES HAD A SPREAD OF 98%. (C) I N CONCLUSION , FREQUENCY , WAVEFORM SHAPE AND SIGNAL ENERGY HAVE A SIGNIFICANT EFFECT ON AF - FECTIVE RESPONSE . T HIS CAN BE GENERALISED ACROSS PARTICIPANTS FOR A ROUSAL , WHEREAS V A - LENCE IS MORE SUBJECTIVE TO THE INDIVIDUAL . P ARTICIPANTS WERE RELIABLY ABLE TO DIFFERENTIATE BETWEEN STIMULI . H OWEVER , RESPONSES VARIED DRAMATICALLY BETWEEN INDIVIDUALS FOR EACH SPE - CIFIC STIMULUS . Participant Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Correlation Coefficient -1 -0.5 0 0.5 1 Arousal vs log 2 (Frequency) Valence vs log 2 (Frequency) Valence -1 -0.5 0 0.5 1 Arousal -1 -0.5 0 0.5 1 Participant Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Correlation Coefficient -0.5 0 0.5 Arousal vs Amplitude Valence vs Amplitude Valence -1 -0.5 0 0.5 1 Arousal -1 -0.5 0 0.5 1 Time (s) 0 0.5 1 1.5 Amplitude (V) -4 -2 0 2 4 Time (s) 0 0.5 1 1.5 Amplitude (V) -4 -2 0 2 4 Time (s) 0 0.5 1 Amplitude (V) -1 -0.5 0 0.5 1 њ = [0, -1.5, 7, 1.5] Time (s) 0 0.5 1 Amplitude (V) -1 -0.5 0 0.5 1 њ = [7, -7, 1.5, 0] C IRCUMPLEX M ODEL OF A FFECT VALENCE AROUSAL Time (s) 0 0.5 1 1.5 2 Envelope amplitude (V) -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 њ = -7 њ = -1.5 њ = 0 њ = 1.5 њ = 7 Stimulus Number 1 2 3 4 5 6 7 Mean response across participants -1 -0.5 0 0.5 1 Music Tactile Combination Participant Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Mean response across stimuli types -1 -0.5 0 0.5 1 Music Tactile Combination A F E B C D 0 50 100 150 200 250 300 time (sec) -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 amplitude 0 5 10 15 20 25 30 35 40 1 2 3 4 5 6 7 8 10 Motor number 0 5 10 15 20 25 30 35 40 time (sec) 40 45 50 55 60 65 70 75 80 85 note number R AW AUDIO FILE M IDI FILE
Transcript of Feel the Music€¦ · feel the music 1experiment a formal experiment was conducted to prototype 2...
EXPLORING TACTILE MAPPINGS OF AUDITORY STIMULI AS A STEP TOWARD MAKING MUSIC MORE ACCESSIBLE FOR THOSE THAT ARE DEAF OR HEARING IMPAIRED.
Feel the Music
PROTOTYPE 2EXPERIMENT1 A FORMAL EXPERIMENT WAS CONDUCTED TO INVESTIGATE OUR EMOTIONAL RESPONSES TO TACTILE STIMULI AND IDENTIFY WHICH FEATURES OF HAPTIC SENSATIONS AFFECT THE USER'S RESPONSE.
RESULTS
STIMULI• 42 STIMULI WERE SINE AND SQUARE WAVES WITH VARYING FREQUENCY AND AMPLITUDE. • 60 STIMULI WERE GENERATED BY MULTIPLYING A 64HZ SINE WAVE WITH AN ENVELOPE WAVEFORM DEFINED BY A PA-RAMETER α. • 6 REPEATED STIMULI.
• 20 PARTICIPANTS TOOK PART. • 108 TACTILE STIMULI WERE PRESENTED.• STIMULI WERE HAPTIC SENSATIONS GENERATED BY A TACTOR PLACED ON EACH PARTICIPANT’S FINGERTIP.
PARTICIPANTS RATED THE STIMULI BY AROUSAL (INTENSITY) AND VALENCE (PLEASANTNESS). THESE MEASURES ARE BASED ON THE CIRCUMPLEX MODEL OF AFFECT, A WIDELY USED MODEL IN PSYCHOLOGY RESEARCH TO MEASURE EMOTIONAL RESPONSES.
EXAMPLE STIMULI PROGRESSION OF PROTOTYPES
TACTORS
α VALUES FOR ENVELOPESTIMULI
RESULTS
THE RESULTS LED TO THE CREATION OF A WEARABLE PROTOTYPE THAT EMPLOYED TACTILE SENSATIONS AS A MUSICAL INTERFACE. MAPPINGS FROM MUSIC SAMPLES ONTO THE DEVICE WERE DESIGNED AND TESTED IN A SECOND EXPERIMENT.
BY: ALICE HAYNESSUPERVISORS: DR J. LAWRY,
DR J. ROSSITER
EXAMPLE TACTILE MAPPING FOR
“ASHES TO ASHES’
AFTER MULTIPLE ITERATIONS AND MODIFICATIONS THE FINAL PROTO-TYPE CREATED WAS A SLEEVE CONTAINING:
• 8 VIBRATION MOTORS WITH SMOOTH CASINGS PLACED IN SEQUENCE ALONG THE FOREARM.• A SQUEEZER MODULE FORMED OF AN ENCLOSED POCKET FILLED WITH AIR VIA A VACUUM PUMP ON THE UPPER ARM.
THESE WERE CONTROLLED BY AN ARDUINO LEONARDO WITH 3 STACKED MOTOR SHIELDS AND 1 SD CARD SHIELD ONTO WHICH MAP-PINGS WERE SAVED. TACTILE MAPPINGS WERE PLAYED IN SYNC WITH MUSIC BY SENDING A START COMMAND VIA SERIAL FROM MATLAB AND PLAYING THE MUSIC SIMULTANEOUSLY.
MAPPINGSCHOSEN MUSIC SAMPLES WERE CONVERTED INTO MIDI FILES FROM WHICH THE KEY MELODIES AND RHYTHMS WERE IDENTIFIED AND TRANSLATED INTO AN ARRAY OF INTENSITIES TO CONTROL THE MOTORS AND SQUEEZER OVER TIME. 7 MUSIC SAMPLES WERE USED, INCLUDING THE JAWS THEMTUNE, ‘IS THIS LOVE’ BOB MARLEY AND ‘ASHES TO ASHES’ DAVID BOWIE.
TESTING• 20 PARTICIPANTS TOOK PART. • 28 STIMULI WERE PRESENTED.• STIMULI INCLUDED MUSIC SAMPLES, PURE TACTILE SENSATIONS AND SYNCRONISED COMBINATIONS OF THE TWO.
PARTICIPANTS RATED STIMULI BY VALENCE (PLEASANTNESS) AND SE-LECTED ADJECTIVES FROM A LIST TAKEN FROM HENVER’S ADJECTIVE CIRCLE TO DESCRIBE THE SENSATIONS.
• MEAN VALENCE RESPONSES ACROSS ALL PARTICIPANTS WERE STA-TISTICALLY THE SAME FOR MUSIC ONLY STIMULI AND COMBINATION STIMULI. (F)• MEAN VALENCE RESPONSES TO TACTILE ONLY STIMULI WERE STA-TISTICALLY LOWER THAN FOR THE STIMULI INCLUDING MUSIC.• PER INDIVIDUAL SONG SAMPLE PREFERENCE FOR MUSIC, TACTILE OR COMBINRTION PRESENTATIONS VARIED. (E) • OVER 60% OF RESPONSES PER SONG SAMPLE WERE STATISTICALLY SIMILAR FOR MUSIC ONLY AND TACTILE ONLY STIMULI SUGGESTING THAT MAPPINGS WERE EFFECTIVE.• AGE OF PARTICIPANTS DID NOT AFFECT THEIR RESPONSE TO MUSIC, BUT OLDER PARTICIPANTS GAVE STATISTICALLY LOWER VALENCE RANK-INGS TO TACTILE STIMULI THAN YOUNGER PARTICIPANTS. THIS MAY BE DUE TO CHANGE IN SKIN SENSITIVITY WITH AGE.• ADJECTIVE RESPONSES GENERALLY MATCHED FOR MUSIC AND TAC-TILE MAPPINGS BUT ALSO FOR PURPOSEFULLY NON-MATCHED STIMULI.
IN CONCLUSION, THE PROTOTYPE SUCCESSFULLY DEMON-STRATED THE POTENTIAL FOR TACTILE STIMULATION TO BE USED AS A SENSORY SUBSTITUTE FOR ACOUSTIC MUSIC.
• THE EXPERIMENT FOUND THAT THERE IS A GENERAL NEGA-TIVE TREND BETWEEN AROUSAL AND VALENCE RESPONSES AS DEMONSTRATED BY THE MEAN RESPONSES ACROSS STIMULI. (A)• FREQUENCY HAS A SIGNIFICANT POSITIVE CORRELATION WITH ELICITED AROUSAL RESPONSES. (B)• FREQUENCY IS SIGNIFICANTLY CORRELATED WITH 80% OF PARTICIPANT VALENCE RESPONSES. HOWEVER, 6 WERE POSI-TIVE AND 10 WERE NEGATIVE. (B)• AMPLITUDE IS NOT SIGNIFICANTLY CORRELATED WITH AROUSAL OR VALENCE ALTHOUGH THERE IS A GENERAL POSI-TIVE CORRELATION WITH AROUSAL AND NEGATIVE CORRELA-TION WITH VALENCE. (D)• SQUARE WAVE STIMULI MORE AROUSING THAN SINE WAVE STIMULI WHICH MAY BE DUE TO THE POSITIVE CORRELATION BE-TWEEN ENERGY IN THE STIMULUS SIGNAL AND ELICITED AROUSAL RESPONSE. • ENERGY IN A STIMULUS SIGNAL ALSO NEGATIVELY CORRE-LATED WITH VALENCE RESPONSES.• REPEATED STIMULI HAD A MEAN SPREAD OF RESPONSES ACROSS 15.5% OF THE SPACE PER PARTICIPANT. THIS INDI-CATED RELIABILITY IN PARTICIPANT RESPONSES SINCE TESTS RUN FOR MULTIPLE RANDOM RESPONSES HAD A SPREAD OF 98%. (C)
IN CONCLUSION, FREQUENCY , WAVEFORM SHAPE AND SIGNAL ENERGY HAVE A SIGNIFICANT EFFECT ON AF-FECTIVE RESPONSE. THIS CAN BE GENERALISED ACROSS PARTICIPANTS FOR AROUSAL, WHEREAS VA-LENCE IS MORE SUBJECTIVE TO THE INDIVIDUAL. PARTICIPANTS WERE RELIABLY ABLE TO DIFFERENTIATE BETWEEN STIMULI. HOWEVER, RESPONSES VARIED DRAMATICALLY BETWEEN INDIVIDUALS FOR EACH SPE-CIFIC STIMULUS.
Participant Number1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Cor
rela
tion
Coe
ffici
ent
-1
-0.5
0
0.5
1Arousal vs log2(Frequency) Valence vs log2(Frequency)
Valence-1 -0.5 0 0.5 1
Arou
sal
-1
-0.5
0
0.5
1
Participant Number1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Cor
rela
tion
Coe
ffici
ent
-0.5
0
0.5
Arousal vs Amplitude Valence vs Amplitude
Valence-1 -0.5 0 0.5 1
Aro
usal
-1
-0.5
0
0.5
1
Time (s)0 0.5 1 1.5
Ampl
itude
(V)
-4
-2
0
2
4
Time (s)0 0.5 1 1.5
Ampl
itude
(V)
-4
-2
0
2
4
Time (s)0 0.5 1
Am
plitu
de (V
)
-1
-0.5
0
0.5
1 = [0, -1.5, 7, 1.5]
Time (s)0 0.5 1
Am
plitu
de (V
)
-1
-0.5
0
0.5
1 = [7, -7, 1.5, 0]
CIRCUMPLEX MODEL OF AFFECT
VALENCEAR
OU
SAL
Time (s)0 0.5 1 1.5 2
Enve
lope
am
plitu
de (V
)
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1 = -7
= -1.5
= 0
= 1.5
= 7
Stimulus Number1 2 3 4 5 6 7
Mea
n re
spon
se a
cros
s pa
rtic
ipan
ts
-1
-0.5
0
0.5
1Music Tactile Combination
Participant Number1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Mea
n re
spon
se a
cros
s st
imul
i typ
es
-1
-0.5
0
0.5
1Music Tactile Combination
A
F
E B
CD
0 50 100 150 200 250 300time (sec)
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
ampl
itude
0 5 10 15 20 25 30 35 40
12345678
10
Mot
or n
umbe
r
0 5 10 15 20 25 30 35 40time (sec)40
45
50
55
60
65
70
75
80
85
note
num
ber
RAW AUDIO FILE
MIDI FILE
