Recent Research in Musical Timbre Perception
James W. BeauchampUniversity of Illinois at Urbana-
Champaign
Andrew B. HornerHong University of Science and
Technology
Michael D. HallJames Madison University, Harrisonburg,
VA
Starting Point• Timbre experiments are based on musical
instrument sounds.
Starting Point• Timbre experiments are based on musical
instrument sounds.• Perform short-time spectral analysis.
Starting Point• Timbre experiments are based on musical
instrument sounds.• Perform short-time spectral analysis.• Identify parameters of ST spectrum:
Starting Point• Timbre experiments are based on musical
instrument sounds.• Perform short-time spectral analysis.• Identify parameters of ST spectrum:
Partial (harmonic) amplitudes- Time variation- Spectral envelope (centroid, irregularity, etc.)
Starting Point• Timbre experiments are based on musical
instrument sounds.• Perform short-time spectral analysis.• Identify parameters of ST spectrum:
Partial (harmonic) amplitudes- Time variation- Spectral envelope (centroid, irregularity, etc.)
Partial (harmonic) frequencies- Time variation- Inharmonicity
Methods for Studying TimbreStimuli Preparation
In Freq. Domain– Simplification– Perturbation– Normalization
Methods for Studying Timbre Listener Experiments
–Discrimination (pairs)–Timbral Distance Estimation–Classification–Identification
Stimuli PreparationIn Freq. Domain
– Simplification– Perturbation– Normalization
Methods for Studying Timbre Listener Experiments
–Discrimination (pairs)–Timbral Distance Estimation–Classification–Identification
Stimuli PreparationIn Freq. Domain
– Simplification– Perturbation– Normalization
Data Processing/Presentation–Discrimination (sensitivity) scores/plots–Multidimensional Scaling–Correspondence (R2) Measurements
Studies Reviewed
• 1999 Discrimination Study
• 2006 Discrimination Study
• 2006 Multidimensional Scaling (MDS) Study
• 2009 Discrimination/Classification Study
1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA)
• Seven reference sounds– clarinet, flute, oboe, trumpet, violin, harpsichord,
marimba
1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA)
• Seven reference sounds– clarinet, flute, oboe, trumpet, violin, harpsichord,
marimba
• Equalize F0, loudness, and duration.
1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA)
• Seven reference sounds– clarinet, flute, oboe, trumpet, violin, harpsichord,
marimba
• Equalize F0, loudness, and duration.
• Test sounds: Apply six spectrotemporal
simplifications.
1999 Discrimination Study(McAdams, Beauchamp, Meneguzzi, JASA)
• Seven reference sounds– clarinet, flute, oboe, trumpet, violin, harpsichord,
marimba
• Equalize F0, loudness, and duration..
• Test sounds: Apply six spectrotemporal
simplifications.
• Subjects discriminate between original and
simplified sounds.
1999 Discrimination StudyResults
• Spectral envelope smoothing 96%
• Spectral flux elimination 91%
• Amplitude envelopes smoothing 66%
• Frequency envelopes smoothing 70%
• Freq. envs. harmonic locking 69%
• Frequency variations elimination 71%
DiscrimScore
2006 Discrimination StudyHorner, Beauchamp, and So JAES
• Eight sustained musical instrument tones– bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin
2006 Discrimination StudyHorner, Beauchamp, and So JAES
• Eight sustained musical instrument tones– bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin
• Modified by fixed random transfer function 1−2ε < H( f) <1+ 2ε, ε = εrror lεvεl
2006 Discrimination StudyHorner, Beauchamp, and So JAES
• Eight sustained musical instrument tones– bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin
• Modified by fixed random transfer function–
• F0, loudness, duration, centroid preserved 1−2ε < H( f) <1+ 2ε, ε = εrror lεvεl
2006 Discrimination StudyHorner, Beauchamp, and So JAES
• Eight sustained musical instrument tones– bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin
• Modified by fixed random transfer function–
• F0, loudness, duration, centroid preserved 1−2ε < H( f) <1+ 2ε, ε = εrror lεvεl
Typical spectral envelopes:original
0
500
1000
1500
2000
300 900 1500 2100 2700 3300
frequency (Hz)
20% error
0
500
1000
1500
2000
2500
300 900 1500 2100 2700 3300
frequency (Hz)
40% error
0500
10001500
200025003000
300 900 1500 2100 2700 3300
frequency (Hz)
2006 Discrimination StudyHorner, Beauchamp, and So JAES
• Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to discrimination between original and modified tones.
2006 Discrimination StudyHorner, Beauchamp, and So JAES
• Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to the discrimination data.
• Best results: obtained by relative-amplitude (harmonic) spectral error:
ε rase =1N
Ak tn( ) − A'k tn( )
a
k =1
K
∑
Aka tn( )
k =1
K
∑a
n=1
N
∑ , 0 < a < 3.0
Usually, a =1 or 2
2006 Discrimination StudyHorner, Beauchamp, and So JAES
R2=0.81
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5error level
discrimination
Discrimination vs. error level (ε):
2006 Discrimination Study
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5relative-amplitude spectral error
discrimination
Discrimination vs. rel-amp spec error:
R2=0.90for
a=1.0
Horner, Beauchamp, and So JAES
2006 MDS Study
• Ten sustained musical instrument tones– bassoon, cello, clarinet, flute, horn, oboe, recorder, alto
sax, trumpet, violin
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2006 MDS Study
• Ten sustained musical instrument tones– bassoon, cello, clarinet, flute, horn, oboe, recorder, alto
sax, trumpet, violin
• F0, loudness, duration, attack & decay times, and average spectral centroid are equalized.
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2006 MDS Study
• Ten sustained musical instrument tones– bassoon, cello, clarinet, flute, horn, oboe, recorder, alto
sax, trumpet, violin
• F0, loudness, duration, attack & decay times, and average spectral centroid are equalized.
• Two types of tones: static (flux removed) and dynamic (flux retained).
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2006 MDS Study
• Ten sustained musical instrument tones– bassoon, cello, clarinet, flute, horn, oboe, recorder, alto
sax, trumpet, violin
• F0, loudness, duration, attack & decay times, and average spectral centroid are equalized.
• Two types of tones: static (flux removed) and dynamic (flux retained).
• Subjects estimate timbral dissimilarity between instruments.
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2006 MDS Study
• Ten sustained musical instrument tones– bassoon, cello, clarinet, flute, horn, oboe, recorder, alto
sax, trumpet, violin
• F0, loudness, duration, attack & decay times, and average spectral centroid are equalized.
• Two types of tones: static (flux removed) and dynamic (flux retained).
• Subjects estimate timbral dissimilarity between instruments.
• Data processed by two multi-dimensional scaling (MDS) programs (SPSS & Matlab).
Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
Acoustical Correlates to Test:• Even/Odd: Ratio of even and odd harmonic rms
amplitudes.
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
Acoustical Correlates to Test:• Even/Odd: Ratio of even and odd harmonic rms
amplitudes
• Spectral IRregularity: Degree of jaggedness of a spectrum.
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
Acoustical Correlates to Test:• Even/Odd: Ratio of even and odd harmonic rms
amplitudes
• Spectral IRregularity: Degree of jaggedness of a spectrum.
• Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value.
For Dynamic Tones Only:
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
Acoustical Correlates to Test:• Even/Odd: Ratio of even and odd harmonic rms
amplitudes
• Spectral IRregularity: Degree of jaggedness of a spectrum.
• Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value.
• Spectral INcoherence: Degree of spectral change relative to the average spectrum (same as flux).
For Dynamic Tones Only:
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2D MDSResults:
Static ToneCase
SPSSalgorithm
Correlations: E/O: R=0.78 SIR: R=0.69
Stress=0.12
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2D MDSResults:
Static ToneCase
Matlabalgorithm
Correlations: E/O: R=0.79 SIR: R=0.75
Stress=0.12
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2D MDSResults:
Dynamic ToneCase
SPSSalgorithm
Correlations: E/O: R=0.71 SCV:R=0.68 SIN: R=0.56 SIR: R=0.39
Stress=0.17
2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)
2D MDSResults:
Dynamic ToneCase
Matlabalgorithm
Correlations: E/O: R=0.69 SCV:R=0.68 SIN: R=0.53 SIR: R=0.40
Stress=0.15
2009 Study Hall and Beauchamp (Canadian Acoustics)
Goals/Purpose• Exp. 1. Relative importance of spectral vs. temporal
cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.
2009 Study Hall and Beauchamp (Canadian Acoustics)
Goals/Purpose• Exp. 1. Relative importance of spectral vs. temporal
cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.
• Exp. 2. Relative importance of spectral envelope (formant) structure vs. spectral centroid: Compare discrimination/classification performance for interpolated tones vs. tones obtained by filtration which matches the centroids of the interpolated tones.
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 1 Method• Reference stimuli: Impoverished (static) violin and trombone
sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 1 Method• Reference stimuli: Impoverished (static) violin and trombone
sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
• Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres.
Vn I01 I02 I03
I10 I11 I12 I13
I20 I21 I22 I23
I30 I31 I32 Tr
Temporal
Spe
ctra
l
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 1 Method• Reference stimuli: Impoverished (static) violin and trombone
sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
• Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres.
• Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.).
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 1 Method• Reference stimuli: Impoverished (static) violin and trombone
sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)
• Test stimuli: A 44 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs-time envelope between the violin and trombone timbres.
• Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.).
• Subjects’ tasks: - 1) to discriminate tone pairs. - 2) to classify tones as ‘violin’, ‘trombone’, or ‘other’.
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 1 Results
Violin
-1.00
0.001.00
2.00
3.00
4.005.00
6.00
1 2 3
Step Size
Sensitivity (d')
Amplitude Envelope
Spectral Envelope
Both
Trombone
-1.00
0.00
1.00
2.003.00
4.00
5.00
6.00
1 2 3
Step Size
Amplitude Envelope
Spectral Envelope
Both
Discrimination:
Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.
reference stimuli:
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 1 ResultsClassification:
Violin Responses
0.000.100.200.300.400.500.600.700.800.901.00
Vn
VnHybridTrHybrid
Tr
Spectral Envelope
Response Probability
Vn
VnHybrid
TrHybrid
Tr
Trombone Responses
0.000.100.200.300.400.500.600.700.800.901.00
Vn
VnHybrid TrHybrid
Tr
Spectral Envelope
Vn
VnHybrid
TrHybrid
Tr
Amplitude Envelope
Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method• Reference stimulus: Original impoverished violin.
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method• Reference stimulus: Original impoverished violin.
• Test stimuli: - 1) 3 violin tones interpolated with respect to spectral
envelope (steps 1-3) (original violin amp env kept).
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method• Reference stimulus: Original impoverished violin.
• Test stimuli: - 1) 3 violin tones interpolated with respect to spectral
envelope (steps 1-3) (original violin amp env kept).- 2) 3 violin tones low-pass filtered in steps to match the
spectral centroids of the 3 interpolated tones of 1).
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Method• Reference stimulus: Original impoverished violin.
• Test stimuli: - 1) 3 violin tones interpolated with respect to spectral
envelope (steps 1-3) (original violin amp env kept).- 2) 3 violin tones low-pass filtered in steps to match the
spectral centroids of the 3 interpolated tones of 1).
• Subjects’ tasks: Discrimination and classification as in Exp. 1. (Which has the greater effect? Interpolation or filtration?)
2009 Study Hall and Beauchamp (Canadian Acoustics)
Experiment 2 Results
0 . 0 0
0 . 5 0
1 . 0 0
1 . 5 0
2 . 0 0
2 . 5 0
3 . 0 0
3 . 5 0
4 . 0 0
4 . 5 0
5 . 0 0
1 2 3
S t e p S i z e
Sensitivity (d')
F o r m a n t S t r u c t u r e
S p e c t r a l C e n t r o i d
Discrimination:
Classification:Trombone Responses
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Formant Structure Spectral Centroid
Stimulus Type
Identification Percentage
Vn
VnHybrid
TrHybrid
Tr
Violin Responses
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Formant Structure Spectral Centroid
Stimulus Type
Response Probability
Conclusion Summary• 1999 discrimination study:
– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.
Conclusion Summary• 1999 discrimination study:
– Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.
• 2006 discrimination study:– The ability to hear differences between
dynamic tones with matched spectral centroids and randomly altered spectra correlates strongly with relative spectral-amplitude differences.
Conclusions• 2006 MDS study:
– Using centroid and attack/decay normalized tones, there is strong evidence that even/odd ratio and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.
Conclusions• 2006 MDS study:
– Using centroid and attack/decay normalized tones, there is strong evidence that even/odd ratio and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.
• 2009 discrimination/classification study:– Using spectral interpolation with respect to both
spectral and temporal dimensions on impoverished violin and trombone tones:
1) Spectral differences were found to be more important than temporal differences.
2) Detailed spectral differences were much more important than mere spectral centroid differences.
Top Related