Introduction Dr Paul Vickers 1
School of Informatics
CG087 Time-based Multimedia Assets
Introduction to time-based multimedia assets
Dr Paul VickersDr Alf Watson
Introduction Dr Paul Vickers 2
CG087 Time-based Multimedia AssetsSchool of Informatics
Introduction
• Looking at multimedia assets that change over time– Sound– Video
• Sound clearly requires a time dimension to be perceived
• By video we mean either motion video, or sequences of still images that are time-based (e.g. synchronised slide show) or combination of both
Introduction Dr Paul Vickers 3
CG087 Time-based Multimedia AssetsSchool of Informatics
What the module covers
• Sound theory– Physics of sound– Psycho-acoustics
• Sound language– Atmospheric sound– Sonic/musical grammars– Foley art and sound tracks– Sound in games
Introduction Dr Paul Vickers 4
CG087 Time-based Multimedia AssetsSchool of Informatics
What the module covers
• Sound as a communication/interaction medium– Auditory display– Sonification, auralisation– Earcons, auditory icons
• Musical instrument digital interface (MIDI)– Definition & history– Protocols– Input– Controllers & tone generators– Sequencers
Introduction Dr Paul Vickers 5
CG087 Time-based Multimedia AssetsSchool of Informatics
What the module covers
• Combining MIDI and audio– Sampling & loops– Sequencing & mixing
• (Compression and streaming– What is compression?– Types of compression (e.g. WMA, MP3, ATRAC))
Introduction Dr Paul Vickers 6
CG087 Time-based Multimedia AssetsSchool of Informatics
What the module covers
• Video– Digital vs analogue– Filming – video language, managing shooting– Digitising, optimising, compression systems– Synchronising sound to video– Compression & streaming– Time-based control structures
Introduction Dr Paul Vickers 7
CG087 Time-based Multimedia AssetsSchool of Informatics
Who does what
• Dr Paul Vickers: Sound– [email protected]– Pandon 1.21– Tel 243-7614
• Dr Alf Watson: Video– [email protected]– Pandon 1.14– Tel 243-7606
Introduction Dr Paul Vickers 8
CG087 Time-based Multimedia AssetsSchool of Informatics
Assessment
• There is one piece of assessment (no exam)– An asset creation, reporting and discussion task– Will be handed out in week 1– Due in week 14
Introduction Dr Paul Vickers 9
CG087 Time-based Multimedia AssetsSchool of Informatics
Introduction to sound
• What is sound?– If a tree falls in a forest and nobody is there to
hear it, does it make a sound?– Discuss…
• Different from vision– Can only attend to one visual stream at once, yet
can monitor several auditory streams– When you mix sounds from several sources we
still perceive them as separate sources
Introduction Dr Paul Vickers 10
CG087 Time-based Multimedia AssetsSchool of Informatics
Sound perception
• Sound is a construction of the mind• Neural coding & processing of information
from auditory system, integration with information from other sensory systems, and responding to the result is what defines hearing
• Sound is temporal– (what is a sound of zero duration?)– Though it has some spatial characteristics.
Discuss.
Introduction Dr Paul Vickers 11
CG087 Time-based Multimedia AssetsSchool of Informatics
Physical definition
• Vibration of an object produces sound. If we hear the vibration, the sound is audible– Hitting a table causes vibration– Blowing across a bottle causes complex vibrations– Plucking a guitar string causes it to vibrate
• What’s the difference between the sounds made by hitting a table, blowing a trombone, whistling?
• Fourier showed that any vibration can be resolved into a sum of sinusoidal vibrations
Introduction Dr Paul Vickers 12
CG087 Time-based Multimedia AssetsSchool of Informatics
Sinusoids
• Sinusoids (aka sine waves) describe relationships between displacement and time
– http://www.keypress.com/sketchpad/java_gsp/sinewaver.html
Introduction Dr Paul Vickers 13
CG087 Time-based Multimedia AssetsSchool of Informatics
Attributes of sine wave
• Frequency (no. of cycles per second, Hz)• Amplitude (height of wave)• Starting phase
– http://www.udel.edu/idsardi/sinewave/sinewave.html
• What does frequency relate to?• What does amplitude relate to?
Introduction Dr Paul Vickers 14
CG087 Time-based Multimedia AssetsSchool of Informatics
Attributes of sound
• Sound has three physical attributes– Frequency– Intensity– Time
• In music we talk about pitch. Is it the same as frequency?– Concert pitch is where the A above middle C has
a frequency of 440Hz
Introduction Dr Paul Vickers 15
CG087 Time-based Multimedia AssetsSchool of Informatics
Complex sounds
• Most sounds are not pure tones, but combinations of frequencies
• Pure tone:
Introduction Dr Paul Vickers 16
CG087 Time-based Multimedia AssetsSchool of Informatics
Complex sounds
• Square wave
• Triangle wave
Introduction Dr Paul Vickers 17
CG087 Time-based Multimedia AssetsSchool of Informatics
Pitch perception
• All three waves had the same perceived frequency yet the first contained only one sinusoid, whilst the square and triangle waves contained many lower-intensity sinusoids above the fundamental of 250Hz
• Thus, we tend to perceive the pitch of a tone as its fundamental (or its missing fundamental!)– You hear a 100Hz pitch when presented with a stimulus
consisting of the sum of the frequencies of 700, 800, 900, & 1000Hz: all four tones are harmonics of 100Hz
• Pitch of sine waves varies with intensity! Pitch increases as level increases
Introduction Dr Paul Vickers 18
CG087 Time-based Multimedia AssetsSchool of Informatics
Timbre
• What differentiated the sound of the three tones we heard earlier? How would you describe them?
• The quality that differentiates how tones sound is called timbre
• Different instruments have different timbres
Introduction Dr Paul Vickers 19
CG087 Time-based Multimedia AssetsSchool of Informatics
Intensity
• Intensity is an measure of the energy of a signal measured in decibels (dB)– 0dB is the threshold of human hearing
(0.000000000001 W/m2)– 3dB is a doubling of intensity– Pain threshold 120dB-140dB– 90dB (or prolonged exposure to lower levels) can
cause permanent hearing damage
Introduction Dr Paul Vickers 20
CG087 Time-based Multimedia AssetsSchool of Informatics
Power ratios and decibels
P/Pr Ratio expressed
in decibels 1,000,000 60 dB
10,000 40 dB 1,000 30 dB
100 20 dB 10 10 dB 4 6 dB 1 0 dB
¼ (=0.25) -6 dB 1/10 (=0.1) -10 dB
1/100 (=0.01) -20 dB P is the power of the measured sound in watts per square metre Pr is the reference-level power, or 10-12 watts-per-square-metre. Number of decibels is 10 log10(P/Pr)
Introduction Dr Paul Vickers 21
CG087 Time-based Multimedia AssetsSchool of Informatics
Example intensities
Description Intensity
Threshold of hearing 0 dB
Outdoors, no people or traffic 20 dB
Background noise in a library 40 dB
Ordinary speaking voice 60 dB
Heavy traffic 85 dB
Power lawnmower up close 90 dB
Motorcycle (rider) 90 dB
Chainsaw 100 dB
Rock music at concert 110 dB
Jet plane takeoff at 100 yards 120 dB
Threshold of pain 140 dB
Introduction Dr Paul Vickers 22
CG087 Time-based Multimedia AssetsSchool of Informatics
Loudness
• Loudness is a psychophysical response which does not necessarily equate to intensity. Usually measured in phons– Phon is the level in dB SPL of an equally loud
1,000Hz tone. All tones judged equal in loudness to a 40dB SPL 1KHz tone have a loudness level of 40 phons.
• Studies have shown loudness doubles every 10dB or so, i.e. sound must be increased in intensity by a factor of ten for the sound to be perceived as twice as loud. Or, it takes 10 violins to sound twice as loud as one violin.
Introduction Dr Paul Vickers 23
CG087 Time-based Multimedia AssetsSchool of Informatics
Equal loudness contours
100Hz, 52dB SPL1KHz, 40 dB SPL4KHz, 37 dB SPLAll judged to be
equal in loudness
Introduction Dr Paul Vickers 24
CG087 Time-based Multimedia AssetsSchool of Informatics
Psychoacoustics
• Psychoacoustics is the psychological study of hearing. The aim of psychoacoustic research is to find out how hearing works.
• We have seen that what we hear is not the same as what is presented
• Why do we hear things in certain ways?
Introduction Dr Paul Vickers 25
CG087 Time-based Multimedia AssetsSchool of Informatics
Auditory scene analysis
• “The things that we see are organized into patterns or figures, rather than discrete dots of light of different colours. In hearing, we also tend to organize sounds into auditory objects or streams. Bregman (1994) has termed this process for audition auditory scene analysis”– Taken from
http://privatewww.essex.ac.uk/~cplack/Courses/PS454_hbk.html
• For info on ASA see http://www.psych.mcgill.ca/labs/auditory/introASA.html
Introduction Dr Paul Vickers 26
CG087 Time-based Multimedia AssetsSchool of Informatics
Other phenomena
• The cocktail party effect (Arons, 1992) allows us to pick out one conversation in a babel of voices
• The tritone paradox (Deutsch, 1991) shows how our language influences the way we hear music– http://www.cameron.edu/~lloydd/webdoc1.html#SECTION3
• Auditory illusions– Shepard risset tones
http://www.exploratorium.edu/exhibits/highest_note/ex.about.fr.html
– And many others. See what you can find.
Introduction Dr Paul Vickers 27
CG087 Time-based Multimedia AssetsSchool of Informatics
References
• Arons, B. (1992). “A Review of the Cocktail Party Effect.” Journal of the American Voice I/O Society 12(Jul.): 35-50.
• Bregman, Albert S. (1994). Auditory Scene Analysis: The Perceptual Organization of sound. Cambridge, Massachusetts: The MIT Press.
• Deutsch, D. (1991). “The Tritone Paradox: An Influence of Language on Music Perception.” Music Perception 8(4): 335-347.
Top Related