Introduction to Imaging and Multimedia - Computer Scienceelgammal/classes/cs334/Introduction.pdf ·...

Introduction to Imaging and Multimedia 9/28/16

A. Elgammal, Rutgers 1

Introduction to Imaging and Multimedia

Fall 2016 Ahmed Elgammal

Rutgers University

§  Introduction to Imaging and Multimedia §  This course has two integrated components:

§  Digital Multimedia §  Image Processing and Computer Vision

§  You can think of it as an introduction to multimedia with a special focus on digital imaging and video



§  What is Multimedia ?

What is Multimedia §  When different people mention the term multimedia, they often

have quite different, or even opposing, viewpoints. §  A PC vendor: a PC that has sound capability, a DVD-ROM

drive, and perhaps the superiority of multimedia-enabled microprocessors that understand additional multimedia instructions.

§  A consumer entertainment vendor: interactive smart TV with hundreds of digital channels available, or a cable TV-like service delivered over a high-speed Internet connection.

§  A Computer Science (CS) student: applications that use multiple modalities, including text, images, drawings (graphics), animation, video, sound including speech, and interactivity.

§  Evolving definition…



Multimedia in Computer Science §  Digital Multimedia - Computational Multimedia

Digital Multimedia is the field concerned with computer-controlled integration of text, graphics, images, videos, audio, and any other medium where every type of information can be represented, transmitted and processed digitally.

Digital Multimedia: Historical Perspective §  The word multimedia was coined in the beginning of

the 1990s §  Mainly after the success of digital audio recording on

CDs §  The next anticipated step was to create digital content

involving image, text, video, along with the audio §  Multimedia CD-ROMs (late 90s) : ex, Encyclopedia

Britannica, games with simple graphics, §  Experience only limited to a single user interacting

with a PC §  Things have changed dramatically since then.



Example MM Applications §  Multimedia involves multiple modalities of text, audio, images,

drawings, animation, and video. Examples of how these modalities are put to use: §  Video teleconferencing. §  Distributed lectures for higher education. §  Tele-medicine. §  Co-operative work environments. §  Searching in (very) large video and image databases for target visual

objects. §  “Augmented” reality: placing real-appearing computer graphics and

video objects into scenes. §  Including audio cues for where video-conference participants are

located. Taking into account gaze direction and attention of participants as well

Example MM Applications §  Building searchable features into new video, and enabling

very high to very low-bit-rate use of new, scalable multimedia products.

§  Making multimedia components editable. allow the user side to decide what components, video, graphics, etc., are actually viewed; allow the client to move components around or delete them. Making components distributed.

§  Building “inverse-Hollywood” applications that can recreate the process by which a video was made. This then allows storyboard pruning and concise video summarization.

§  Using voice-recognition to build an interactive environment, say a kitchen-wall web browser



Multimedia life cycle §  Four main phases in multimedia content life cycle

§  Multimedia Generation, Authoring, Capturing… §  Multimedia Representation, and processing §  Multimedia Retrieval: answering user queries. §  Multimedia Delivery

Capturing (digitization)

Authoring Editing

Indexing Archival

Client

Retrieval

Sens

ors /

inpu

t

MM Delivery

MM Generation MM Archival MM Retrieval

User ?

User ?

User ?

User ?



Inherent Quality of Multimedia Data §  Digital: it is always bits and bytes §  Voluminous: the storage and transmission bandwidth

limitations require that the data be compressed §  Interactive §  Real-time and synchronization: very small and

bounded delay while transmitting information. Intra-media and inter-media synchronization

§  Multimedia and Computer Science: Multimedia is in the

intersection among different areas §  Graphics, §  HCI, §  Visualization, §  Computer vision, §  Data compression, §  Algorithms, §  Networking, §  Database systems, §  Data mining §  Architecture and operating systems, §  ….



Multimedia Research Topics and Projects To the computer science researcher, multimedia consists of a wide

variety of topics: §  Multimedia processing and coding: multimedia content

analysis, content-based multimedia retrieval, multimedia security, audio/image/video processing, compression, etc.

§  Multimedia system support and networking: network protocols, Internet, operating systems, servers and clients, quality of service (QoS), and databases.

§  Multimedia tools, end-systems and applications: hypermedia systems, user interfaces, authoring systems. Multi-modal interaction and integration: “ubiquity” web-everywhere devices, multimedia education including Computer Supported Collaborative Learning and design, and applications of virtual environments.

Multimedia Systems §  A Multimedia System is a system capable of

processing multimedia data and applications. §  A Multimedia System is characterized by the

processing, storage, generation, manipulation and rendition of Multimedia information.



Components of a Multimedia System

§  Capture devices §  e.g. Video Camera, Microphone, Digitising/Sampling Hardware, etc.

§  Storage Devices §  e.g. Hard disks, CD-ROMs, DVD, Blu-ray, etc

§  Communication Networks §  Internet, wireless internet, etc.

§  Computer Systems §  e.g. Desktop machines, Workstations, smart phones, tablets

§  Rendering Devices §  e.g. CD-quality speakers, HDTV, Hi-Res monitors, Color printers etc.



Challenges for Multimedia Systems §  Distributed Networks §  Temporal relationship between data

§  Render different data at same time continuously. §  Sequencing within the media (e.g. playing frames in correct

order/time frame in video) §  Synchronization — inter-media scheduling

§  E.g. Video and Audio — Lip synchronization is clearly important for humans to watch playback of video and audio and even animation and audio.

Hardware §  All contemporary personal computers are quite

capable of displaying MM content §  Even Almost all cell phones §  For authoring content, a more powerful machine is

needed §  Bandwidth is still and always will be an issue



Multimedia System Desirable Features

§  Very High Processing Power §  Special Hardware/Software needed (e.g. GPUs) §  Efficient I/O §  Large Storage and Memory §  High Speed Network Support

Multimedia History §  Before the digital age…



Multimedia: Historical Perspective 3

storage and distribution. The process mostly involves state-of-the-artcompression algorithms and standards for audio, video, images, and graphics.

• Distribution—Distribution involves how multimedia content is distributed viavarious media, such as wired cables, optical networks, satellite, wirelessnetworks, or any combination thereof, to specific platforms ranging fromtelevision, computers, personal digital assistants (PDAs), and so on.

This threefold view is not new and has been used for information in general—creating or gathering information, storing or recording it, and distributing it tothe end user. The table shown in Figure 1-1 gives an evolutionary perspective onthe type of information that people have grown accustomed to through the ages, the

Time and Type of Storage Mode of Age era information medium distribution

Prehistoric 15,000 BC Sounds to Rock –communicate, surfaces,gestures, cave wallspainting

Ancient 500 BC Alphabets, Invention of People drawing paper delivering

messages,horseback

Middle Ages 400–1000 AD Letters, Books Beginning ofwriting a postal

system

Renaissance 1300–1800 AD News, Books, Printing paintings, libraries press, steammagazine engines,

automobiles

Modern world 1900 AD Morse code, Film, Telegramradio, magnetic service,photographs, tapes, wirelessmovies phonograph radio waves

Electronic 1950–1980 Telephone, Electronic Radio and TVtelevision, fax, memory, broadcasting,computers cassette tapes, satellite

LP records communication

Digital 1980 to Computers, Hard disks, Ethernet,present day digital video, CD-ROMs, wireless

surround DVDs networks,sound optical

networks

Figure 1-1 A brief evolution of information

Copyright 2009 Cengage Learning, Inc. All Rights Reserved. May not be copied, scanned, or duplicated, in whole or in part.

Licensed to:

Multimedia: Text

Parchment Papyrus Stone



Paper §  Han Dynasty of China (202 BC). Raw material such as tree bark was finely chopped, mixed

with water, spread onto screens, and dried. Well guarded secret §  Introduced in Europe in 600AD through the Middle East §  First paper mill in Europe was in Spain, in 1120. More mills appeared in Italy in

about the 13th century. They used hemp and linen rags as a source of fiber. Paper is recorded as being manufactured in both Italy and Germany by 1400.

Printing Press

§  In 1451, Johannes Gutenberg and Johannes Fust went into partnership and produced a forty-two line Bible and a thirty-two line Latin Grammar

§  By 1465, printing presses based on Gutenberg's moveable type could be found in Italy, by 1470 in Paris. London followed in 1480. By 1499, there were presses in Stockholm, Constantinople and Lisbon.

§  By 1500, Europe contained in excess of nine million volumes, of thirty thousand titles all of which came off the presses of more than one thousand printers.



Multimedia Data: Text §  Text

§  Input: keyboard, touch pad §  Stored and input character by character. §  Storage of text is 1 or 2 bytes per character. §  Other forms of data (e.g. Spreadsheet files, XML) may store format as

text (with formatting). §  Format: Raw text or formatted text

e.g HTML, Rich Text Format (RTF), Word Processing, or a programming language source

Sequential: may have natural implied sequence e.g. HTML format sequence, Sequence of Java program statements.

§  Size Not significant compared with other multimedia data.

Images

Niepce 1825 (8 hours)

Daguerre 1838 (10 mn)

1888 Eastman's Kodak camera on the market with the slogan "You press the button, we do the rest"

Maxwell 1861 (color)

Now: digital CCD 1969 at AT&T, CMOS

Camera obscura Mozi 400BC Aristotle 350BC Ibn al-Haitham, 1000AD

Painting on some media: From Prehistoric times till today



Multimedia Data: Images §  Images

§  Still pictures which (uncompressed) are represented as a bitmap (a grid of pixels).

§  Input: Digital camera, scanner or generated by graphics editing programs §  Analog sources will require digitizing. §  Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale,

Color Map) or 24 Bits per pixel (True Color) §  Size: a 512x512 Grey scale image takes up 1/4 Mb, a 512x512 24 bit

image takes 3/4 Mb with no compression. §  Storage increases with image size §  Compression is commonly applied

Stored music

15th century

1598

1870 Player piano



Stored audio

1860:phonautograph

Modern period: music

Then digital

1877-1887:

Phonograph and gramophone

1904

Read-Write !



Multimedia Data: Audio §  Audio

§  Audio signals are continuous analog signals. §  Input: microphones and then digitized and stored §  usually compressed. §  CD Quality Audio requires 16-bit sampling at 44.1 KHz §  1 Minute of Mono CD quality audio requires 5 Mb.

Multimedia Data: Video §  Video

§  Input: usually captured by a video camera. §  There are a variety of video (analog and digital) formats §  Raw video can be regarded as being a series of single images. §  There are typically 25, 30 or 50 frames per second. §  a 512x512 size monochrome video images take 25*0.25 = 6.25Mb for 1

second to store uncompressed. §  Digital video clearly needs to be compressed.



How many images and videos are online?

§  Cameras are everywhere now: in our pockets, watching over us at different scales, even inside our bodies. §  Need tools for processing, annotation, archiving,…

§  Images and movies are everywhere: §  In 2009: 2.5 B photos uploaded to Facebook every

month §  That’s 30 B photos per year on FB alone §  In 2011: 6 B per month – 90 B total §  Youtube: in 2011, 48 hours of video are uploaded every

minute, resulting in nearly 8 years of content uploaded every day. In 2014, the number is 100 hours/minute.

Environment

user

Traditional Multimedia Interface vs. Ubiquitous Interface



Multimedia in a Ubiquitous Computing World

§  Multimedia is becoming available all the time around us. §  Ubiquitous computing focus on integrating computation into the

environment, rather than having computers as distinct objects. §  Ubiquitous computing covers wide range of research topics,

such as distributed computing, mobile computing, and sensor networks.

§  Mobile phones now are the most commonly used multimedia devices.

What this course is not about §  This course is not about

§  Web design §  Multimedia tools §  Computer graphics and animation §  Internet technology



Resources §  Multimedia Systems Ch 1 §  Some Slides by Prof G. Medioni @ USC

Introduction to Imaging and Multimedia - Computer Scienceelgammal/classes/cs334/Introduction.pdf ·...

Documents

Transcript of Introduction to Imaging and Multimedia - Computer Scienceelgammal/classes/cs334/Introduction.pdf ·...