Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s...
Transcript of Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s...
![Page 1: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/1.jpg)
Unit I :Information Theory
and Source Coding
Dr. Vandana M. Rohokale
Professor
SITS, Pune
1ITCT & CN-Unit 1_2017-1812/11/2017
![Page 2: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/2.jpg)
Syllabus
• Introduction to information theory
• Entropy and its properties
• Discrete Memory less channels, Mutual information
• Source coding theorem
• Huffman coding• Huffman coding
• Shannon-Fano coding
• The Lempel-Ziv algorithm
• Run Length Encoding
• Examples of Source coding-Audio and Video Compression
2ITCT & CN-Unit 1_2017-1812/11/2017
![Page 3: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/3.jpg)
Introduction to Information Theory
Claude Shannon Found Science of Information theory in
1948
• In his 1948 paper, A Mathematical Theory of
Communication, Claude E. Shannon formulated the
theory of data compression. Shannon established that there is
a fundamental to lossless data compression.
12/11/2017 ITCT & CN-Unit 1_2017-18 3
a fundamental to lossless data compression.
• This limit, called the Entropy Rate, is denoted by H. The
exact value of H depends on the information source --- more
specifically, the statistical nature of the source.
• It is possible to compress the source, in a lossless manner,
with compression rate close to H. It is mathematically
impossible to do better than H.
![Page 4: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/4.jpg)
Information theory is where probability theory goes to work for practical living.
Data Information
Data : how information is represented
12/11/2017 ITCT & CN-Unit 1_2017-18 4
Data : how information is represented
Information : what is represented in data
-- tells us something that we did not already know and would not reliably predict
-- contain a certain element of surprise
![Page 5: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/5.jpg)
• Let’s consider these three sentences for developing mathematical
measure of information
Self Information and Mutual Information
12/11/2017 ITCT & CN-Unit 1_2017-18 5
![Page 6: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/6.jpg)
Mutual Information
12/11/2017 ITCT & CN-Unit 1_2017-18 6
),()()(
)|()(
)|()();(
YXHYHXH
XYHYH
YXHXHYXI
−+=
−=
−=
![Page 7: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/7.jpg)
Entropy
• Shannon used the ideas of randomness and entropy from the study of
thermodynamics to estimate the randomness (e.g. information content or entropy) of
a process.
12/11/2017 ITCT & CN-Unit 1_2017-18 7
![Page 8: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/8.jpg)
Properties of Entropy
12/11/2017 ITCT & CN-Unit 1_2017-18 8
Joint Entropy: H(X,Y) = H(X) + H(Y|X)
![Page 9: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/9.jpg)
Numerical Example
12/11/2017 ITCT & CN-Unit 1_2017-18 9
![Page 10: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/10.jpg)
12/11/2017 ITCT & CN-Unit 1_2017-18 10
![Page 11: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/11.jpg)
Channel Capacity
12/11/2017 ITCT & CN-Unit 1_2017-18 11
![Page 12: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/12.jpg)
Source Coding Theorem
Shannon’s Vision
Example - Disk Storage
12/11/2017 ITCT & CN-Unit 1_2017-18 12
Example - Disk Storage
![Page 13: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/13.jpg)
Example – VCD and DVD
Example – Cellular Phone
12/11/2017 ITCT & CN-Unit 1_2017-18 13
Example – Cellular Phone
![Page 14: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/14.jpg)
Shannon showed:
“To reliably store the information generated by some
random source X, you need no more/less than, on the average,
H(X) bits for each outcome.”
Shannon’s Source Coding Theorem
12/11/2017 ITCT & CN-Unit 1_2017-18 14
![Page 15: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/15.jpg)
12/11/2017 ITCT & CN-Unit 1_2017-18 15
Source- Hsiao-feng Francis Lu, “ Introduction to Information Theory”, National Chung-Cheng Univ
![Page 16: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/16.jpg)
12/11/2017 ITCT & CN-Unit 1_2017-18 16
![Page 17: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/17.jpg)
12/11/2017 ITCT & CN-Unit 1_2017-18 17
![Page 18: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/18.jpg)
Source Coding Theorem
12/11/2017 ITCT & CN-Unit 1_2017-18 18
![Page 19: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/19.jpg)
Shannon Fano Coding
12/11/2017 ITCT & CN-Unit 1_2017-18 19
![Page 20: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/20.jpg)
12/11/2017 ITCT & CN-Unit 1_2017-18 20
![Page 21: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/21.jpg)
Huffman Coding
12/11/2017 ITCT & CN-Unit 1_2017-18 21
![Page 22: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/22.jpg)
The Lempel-Ziv algorithm
12/11/2017 ITCT & CN-Unit 1_2017-18 22
![Page 23: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/23.jpg)
12/11/2017 ITCT & CN-Unit 1_2017-18 23
![Page 24: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/24.jpg)
Run Length Encoding
12/11/2017 ITCT & CN-Unit 1_2017-18 24
![Page 25: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/25.jpg)
12/11/2017 ITCT & CN-Unit 1_2017-18 25
![Page 26: Unit I :Information Theory and Source Coding · 2019. 1. 22. · Source Coding Theorem Shannon’s Vision Example - Disk Storage 12/11/2017 ITCT & CN-Unit 1_2017-18 12. Example –](https://reader034.fdocuments.net/reader034/viewer/2022051813/60347cc18804f540dd3ab969/html5/thumbnails/26.jpg)
26ITCT & CN-Unit 1_2017-18
12/11/2017
Thank You !!!