T.Sharon-A.Frank1

Multimedia

Continuous Media: Audio,

Animation, Video

2

• Conversion of energy into vibrations in the air.

• Most sound sources vibrate in complex ways leading to sounds with components at several different frequencies.

• Frequency spectrum – relative amplitudes of the frequency components.

• Range of human hearing: roughly 20Hz–20KHz, falling off with age.

Sound

T.Sharon-A.Frank

3

Use of Audio

1. Convey Information2. Set a mood3. Capture attention4. Explain a process5. Provide personal contact6. Tell a story

T.Sharon-A.Frank

4

Nature of Sound

• Sound is produced by the vibration of matter.

• Pressure variations are created in air surrounding matter.

• Pattern of oscillation is called a waveform.

5

• Sounds change over time– e.g. musical note has attack and decay, speech

changes constantly.

• Frequency spectrum alters as sound changes.

• Waveform is a plot of amplitude against time:– Provides a graphical view of characteristics of a

changing sound.– Can identify syllables of speech, rhythm of music,

quiet and loud passages, etc.

Waveforms

T.Sharon-A.Frank

6

Basic Sound Concepts

• Waveform:– Period

– Frequency

– Amplitude

• Computer Representation of Sound:– Sampling

– Quantization

T.Sharon-A.Frank

7

Period

• Repeating portion of the sound wave, although sound waves are never perfectly smooth or uniformly periodic.

T.Sharon-A.Frank

8

Frequency

• Reciprocal of the period, that is, number of periods in a second; measured in hertz (Hz) or cycles per second.

• Human hearing frequency range is from 20Hz to 20KHz:

– Infrasound: 0 - 20Hz

– Ultrasound: 20KHz - 1GHz

– Hypersound: 1GHz - 10THz

T.Sharon-A.Frank

9

Amplitude

• Measure of displacement of the air pressure wave from its mean.

• Subjectively heard as loudness.

amplitude

T.Sharon-A.Frank

10

Sound Wave Example

T.Sharon-A.Frank

11

0

0.2

0.4

0.6

0.8

1

1.2

1 4 7 10 13 16 19 22 25 28 31 34 37

Series1

0

0.2

0.4

0.6

0.8

1

1.2

1 4 7 10 13 16 19 22 25 28 31 34 37

Series1

Idea of Audio Digitization

• Smooth continuous curve of waveform is not directly represented in computer.

• Amplitude of waveform is measured and recorded at regular time intervals - process is called encoding.

• Each interval is called a sample.

• Need Analog-to-Digital Converter (ADC).

12

• Converting signal from analogue to digital form

– Analogue signal can vary continuously; digital is restricted to discrete values.

• Two-stage process:

– Sampling – measure the value at discrete intervals.

– Quantization – restrict the value to a fixed set

of quantization levels.

Audio Digitization

T.Sharon-A.Frank

13

Samples

Sample Height

Digital Sampling

Time

Samples

T.Sharon-A.Frank

14

Digital Quantization

Samples

Sample HeightExample: 3 bit quantization

Samples

T.Sharon-A.Frank

15

Audio Encoding

(a) A sine wave. (b) Sampling the sine wave. (c) Quantizing the samples to 4 bits.

T.Sharon-A.Frank

16

Process of Sampling Audio

T.Sharon-A.Frank

17

Sampling Rate

• Number of samples per second measured in hertz (Hz).

• More samples = higher quality = bigger files.

• Example:

– CD quality = 44,100 Hz (44.1 KHz)

– Others: 8,000 Hz, 11,025 Hz, 22,050 Hz

T.Sharon-A.Frank

18

Sampling Size

• Number of bits used to represent each sample.

• Called also Bit Depth and Sound Resolution.

• 8-bit - telephone quality (typical of Web).

• 16-bit - CD quality.

• Examples:

16 bit, 44.1kHz

8 bit , 44.1KHz

Web quality (11.025 KHz, 8bit)

T.Sharon-A.Frank

19

Channels

• Audio files can support from 1 to 6 channels of audio information.

• Most common:– mono (1 channel)

– stereo (2 channels)

• 4-6 channels are usually used for surround sound.

T.Sharon-A.Frank

20

Record Settings Menu for Audio

T.Sharon-A.Frank

21

Sample Audio File Formats

• aif/aiff

• wav

• au

• mov

• MPEG/MP3 (MPEG 1 Level 3)

• RA/RAM• MIDI (Musical Instrument Digital Interface)

T.Sharon-A.Frank

22

Audio File Formats (platform specific)

• aif (aiff)– Audio Interchange File Format– original audio format for Mac– supported by Windows and other platforms

• wav– Waveform Audio File Format– original Windows audio file format– supported on Mac

• au – u-law (myoo-law) – Unix standard audio format

T.Sharon-A.Frank

23

Audio File Formats (container formats)

• mov (qt)– QuickTime, not just Apple/MAC

• mpg (mpe, mpeg)– Moving Pictures Expert Group– High quality, excellent compression file format– Files tend to be small and sound good

• ra (ram)– RealAudio

T.Sharon-A.Frank

24

Real Audio Format

• First example of streaming audio on the Web.

• Quality like that of an AM radio station.

• Best used for narration and not for music or other sounds.

• Must have RealPlayer on your system to listen to files.

25

• MIDI = Musical Instruments Digital Interface

• Instructions about how to produce music, which can be interpreted by suitable hardware and/or software.

• Standard protocol for communicating between electronic instruments (synthesizers, samplers, drum machines).

• Allows instruments to be controlled by hardware or software sequencers.

MIDI

26

• MIDI interface allows a computer to send MIDI data to instruments.

• Store MIDI sequences in files, exchange them between computers, incorporate them into multimedia.

• Computer can synthesize sounds on a sound card, or play back samples from disk in response to MIDI instructions– Computer becomes primitive musical instrument

(quality of sound inferior to dedicated instruments).

MIDI and Computers

T.Sharon-A.Frank

27

Uncompressed Audio Digital Telephony

• Sampled at 8KHz (8K cycles per second)

• Quantized to 8 bits per sample (1 Byte)

=> 64,000bits/s

• Bandwidth = 64,000/1,024 =

Total: 62.5Kb/s

T.Sharon-A.Frank

28

Uncompressed Stereo Audio CD Quality

• Sampled at 44.1KHz (44.1K cycles per second)

• Quantized to 16 bits per sample (2 bytes)

• Two channels for stereo

• Bandwidth =

2 * 44,100 * 16 = 1,411,200b/s

1,411,200/1024= 1,378Kb/s

Total: ~1.4Mb/s

29

• The creation of moving pictures one frame at a time:

– Literally 'to bring to life‘.

– e.g. make a sequence of drawings on paper, in which a character's position changes slightly in each drawing.

– Photograph drawings in sequence, using movie camera that advances one frame at a time.

– Play back resulting film – character will move.

Animation

T.Sharon-A.Frank

30

Use of Animation

1. Capture the viewer’s attention, entertain, or promote.

2. Explain a system or process.

3. Set a mood or create an on-screen environment.

T.Sharon-A.Frank

31

Animation Principles

• Persistence of vision:– object seen by human eye remains mapped on

retina for a brief time after viewing.– display series of images rapidly and they blend

together to create illusion of movement.– requires at least 16 frames per second (fps) to look

seamless.

• Television: displayed at 30 frames per second.

• Movies: displayed at 48 frames per second.

T.Sharon-A.Frank

32

Cell Animation (Frame Animation)

• Start with first and last image in motion (called keyframes).

• Draw images between keyframes (process called tweening/morphing) - small changes between images.

• Images are then layered onto background scene.

• Images displayed rapidly

– 15/24/30 frames per second.

33

• Logic and procedural concepts are same as in cell animation (keyframe, tweening, layering techniques).

• But use computer and video camera.

• Frame grabbing – record each frame to disk.

• Edit non-linearly like for video.

• Can also use scanner or digital still camera, or create each frame in a graphics program (e.g., Painter).

Captured Animation

T.Sharon-A.Frank

34

Cell-based Techniques

• Morphing:– uses frames to create illusion of one object

changing into another– more keyframes = smoother morph.

• Need at least 15 frames per second.• Professional quality uses 24-30 frames per second.• Cell-based animation is done more easily on computer

than by hand.• Easier to create and process than movies.

T.Sharon-A.Frank

35

Path-based Animation (Vector Animation)

• Create animated objects by following object’s transition over a line or vector.

• Artist creates one drawing and a path.

• Computer program manipulates object by drawing frames as object travels over the path.

T.Sharon-A.Frank

36

Sample Animation File Formats

• Animated GIF

• mov (QuickTime)

• SWF (Flash)

• PNG (PortableNetwork Graphics)

37

• Sequence of images can be stored in a single GIF file, and displayed one after another by a Web browser or other software:

– No browser plug-in required

– Can specify looping, delay between frames

– 256 colour palette

– No sound.

Animated GIFs

38

• Popular Web animation format.

• Usually generated by Macromedia Flash.

• Vector animation format

– Motion represented as numerical operations on vector data.

• Can also include bitmapped images (e.g., as backgrounds).

SWF

T.Sharon-A.Frank

39

Video

• Video is sequence of images displayed at constant rate.

• Rate is in frames per second (fps).

• Needs around 10-15 fps for smooth motion.

• For example, video is displayed at 24-30 fps.

T.Sharon-A.Frank

40

Use of Video

1. Tell a story

2. Entertain an audience

3. Explain a complex process

4. Personalize a Multimedia experience

T.Sharon-A.Frank

41

Movie - Characteristics

• Built from Frames - Each frame is an Image– Image characteristics (i.e., resolution, colors)

Clip

Scene

ShotFrame Frame

T.Sharon-A.Frank

42

A movie may consist of several files

T.Sharon-A.Frank

43

Sample Video File Formats

• MOV

• AVI

• RA/RAM

• ASF

• WMV

• MPEG1/2/4

• FLV

T.Sharon-A.Frank

44

MPEG

• MPEG = Moving Picture Experts Group.• Family of compression algorithms.• High compression rate.• Algorithm stores only changes from one frame

to another rather than each entire frame.• Video info then encoded using DCT.• Lossy compression but loss of data is generally

imperceptible to human eye.

T.Sharon-A.Frank

45

MPEG Video Format

• ISO/IEC MPEG (Moving Picture Experts Group) Series of standards including:

1. MPEG-1: used for video CD 352x240 @30 frames/second.

2. MPEG-2: used for compressing DVD and high-definition television (HDTV).

3. MPEG-4: used to transmit audio, video, and graphics at low bitrate – can be delivered over very slow connections (56 Kbps).

T.Sharon-A.Frank

46

Video Acquisition

• TV set and PC monitor look similar.• TV is analog

– Gets info from continuously varying broadcasting waves.

• PC is digital– Data displayed is digital.

• Digitization performed by video capture card.• Typical card is combination of hardware and software.• Capture card typically creates uncompressed files that

then must be passed into a codec chip or software.

T.Sharon-A.Frank

47

Issues Affecting Video File Size

• Movie Length - simple principle: shorter video clips result in smaller files.

• Frame Size - full-screen video is 640x480. Typical frame size on WWW = 160x120 although can handle 320x240.

• Frame Rate:– TV quality = 30 fps – Web = 15 (or even 10) fps.

T.Sharon-A.Frank

48

Video Presentation

• TV– PAL

– NTSC

– SECAM

– HDTV

• Computer Display

49

• NTSC (National Television Standard Committee)

– Used by TV and video industries in US and Japan Taiwan, parts of South America, …

• PAL (Phase Alternating Line)– Used in Western Europe, Australia & New Zealand,

China , …

• SECAM (Sequential Color and Memory)– France and former Soviet Union– Standard only used for transmission– Very different technology than PAL.

TV Standards

T.Sharon-A.Frank

50

Video (PAL, HDTV) Bandwidth

• PAL Resolution: 640 x 480

• 3 bytes per pixel

• Frame rate: 25 fps

• Bandwidth:

640 * 480 * 25 * 24 = 184,320,000b/s

Total: ~176Mb/s

HDTV bandwidth increased by factor of 5.33: ~936Mb/s

51

• PAL uncompressed

– 768x576 pixels per frame

• x 3 bytes per pixel

• x 25 frames per second

≈ 31 MB per second

≈ 1.85 GB per minute

More PAL and NTSC Examples

• NTSC uncompressed

– 640x480 pixels per frame

• x 3 bytes per pixel

• x 30 frames per second

≈ 26 MB per second

≈ 1.6 GB per minute