Class 28: Entropy
-
Upload
david-evans -
Category
Technology
-
view
300 -
download
0
description
Transcript of Class 28: Entropy
![Page 1: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/1.jpg)
Class 28: Entropy
cs1120 Fall 2011David Evans28 October 2011
A giant python swallows an alligator in Everglades National Park, Fla..Paul Zenk/PBS Nature “Invasion of the Giant Pythons” (yesterday’s New York Times)
![Page 2: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/2.jpg)
2
Plan
Story so Far: Liberal Arts RecapReversing in PythonShannon’s Information Theory: EntropyReturning Quiz 3, PS5, etc.
![Page 3: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/3.jpg)
Course RoadmapSynthesis
Analysis
Ch 2
: Lan
guag
e
Ch 3
: Pro
gram
min
g
Ch 4
: Pro
cedu
res
Ch 5
: Dat
aCh
6: M
achi
nes
Ch 7
: Cos
t
Ch 8
: Sor
ting
and
Sear
chin
g
PS5,
Ch
9: S
tate
PS6,
Ch
10: O
bjec
ts
Intr
acta
bilit
y
PS7,
Ch
11: I
nter
pret
ers
Ch 1
2: C
ompu
tabi
lity PS
8, 9
You
are
here
Ch 1
: Com
putin
g
![Page 4: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/4.jpg)
cs1120 Main Topics
Language: How to describe information processes by defining procedures (Chapters 3, 4, 5, 9, 10)Programming with procedures, lists, recursion (PS1-4)Programming with state (PS5), objects (PS6), languages (PS7)
Logic: How to predict properties about information processes (Chapter 6, 7, 8)Predicting how running time grows with input sizeAre there problems which can’t be solved by algorithms? (Ch 12)
Machines: How to efficiently implement information processes (Chapter 6) (not much on this)Machines, Digital LogicHow to implement a Scheme interpreter (Ch 11, PS7)
![Page 5: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/5.jpg)
From Lecture 1:
![Page 6: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/6.jpg)
• Grammar: study of meaning in written expression
• Rhetoric: comprehension of verbal and written discourse
• Logic: argumentative discourse for discovering truth
• Arithmetic: understanding numbers
• Geometry: quantification of space
• Music: number in time
• Astronomy
BNF, RTN, rules of evaluation for meaning
Interfaces between components (PS6),
evaluation rules
Rules of evaluation, if, recursive definitions
adding with logic,representing numbers
Curves as procedures, fractals (PS3)
Ada, GEB Chapter
Nothing!
Triv
ium
Qua
driv
ium
But…read the Neil DeGrasse Tyson essay!
![Page 7: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/7.jpg)
7
Lists (Tuples) in Python
Scheme (list)> (define p (list 1 2 3))> (car p)1> (cdr p)(2 3)> (cdr (cdr (cdr p)))null
Python (tuple)>>> p = (1, 2, 3)>>> p[0]1>>> p[1:](2, 3)>>> p[1:][1:][1:]()
![Page 8: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/8.jpg)
Python Shortcuts
>>> p = (1, 2, 3)>>> p[2]3>>> p[3:]()>>> p[:2](1, 2)
list[i] Evaluates to ith (counting from 0) element of tuple/list
Runs in (approximately) constant time!
![Page 9: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/9.jpg)
9
Mutable Lists (Lists) in Python
Scheme (mutable list)> (define p (mlist 1 2 3))> (mcar p)1> (mcdr p)(2 3)> (mcdr (mcdr (mcdr p)))null
Python [List]>>> p = [1, 2, 3]>>> p[0]1>>> p[1:](2, 3)>>> p[1:][1:][1:]()
![Page 10: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/10.jpg)
Using Mutable Lists
>>> p = [1, 2, 3]>>> p[0]1>>> p[0] = 2>>> p[2, 2, 3]>>> p.append(4)>>> p[2, 2, 3, 4]
Note: this is different from mappend!It take a single element, not a list.
Scheme Application:(<verb> <operands>)Python procedure call:<verb>(<operand>)Method call:<object>.<verb>(<parameters>)
![Page 11: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/11.jpg)
Trick Question
>>> p = (1, 2, 3)>>> p[0] = 2Traceback (most recent call last): File "<pyshell#23>", line 1, in <module> p[0] = 2TypeError: 'tuple' object does not support item assignment
![Page 12: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/12.jpg)
Reversing a Mutable List
1
p:
2 3
Scheme:
1 2 3
p:
Python:
![Page 13: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/13.jpg)
mlist-reverse!
(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))
![Page 14: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/14.jpg)
Literal Translation(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))
![Page 15: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/15.jpg)
Literal Translation
>>> p=[1,2,3]>>> sreverse(p)>>> p[3, 2, 1]
def sreverse(p): if len(p) == 1: return else: rest = p[1:] first = p[0] sreverse(rest) p[0] = rest[0] p[1:] = rest[1:] p.append(first)
(define (mlist-reverse! p) (if (null? (mcdr p)) (void) (let ((rest (mcdr p)) (carp (mcar p))) (mlist-reverse! rest) (set-mcar! p (mcar rest)) (set-mcdr! p (mcdr rest)) (mlist-append! p (mcons carp null)))))
This is correct, but no sane Python programmer would do it this way!
![Page 16: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/16.jpg)
Python for
Statement ::= for Varible in Expression: Block def gaussSum (n): sum = 0 for i in range(1, n+1): sum = sum + i return sum
(define (loop index result test update proc) (if (test index) (loop (update index) (proc index result) test update proc) result)) (define (gauss-sum n) (loop 1 0 (lambda (i) (<= i n)) (lambda (i) (+ i 1)) (lambda (i sum) (+ i sum))))
![Page 17: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/17.jpg)
Reverse in Python
1 2 3
p:
Note: there is a built-in list method, p.reverse().
![Page 18: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/18.jpg)
def reverse(p): for i in range(0, len(p) >> 1): p[i], p[len(p) – 1 - i] = p[len(p) – 1 - i], p[i]
int(len(p) / 2) len(p) >> 1
![Page 19: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/19.jpg)
19
Information Theory
![Page 20: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/20.jpg)
20
Entropy
H = -pi log2 pi
![Page 21: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/21.jpg)
21
(Ideal) Coin Toss
H = -pi log2 pi
flickr cc:Daniel Dale
![Page 22: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/22.jpg)
22
Nucleotide
N {A, C, G, T}Intuition: need two bits to distinguish four possible values
00 = A, 01 = C, 10 = G, 11 = T
H = -pi log2 pi
![Page 23: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/23.jpg)
23
![Page 24: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/24.jpg)
24
Claude Shannon’s Juggling W. C. Fields
Real Coin Toss?
![Page 25: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/25.jpg)
25
Real Coin TossAccording to Susan Holmes, 51% of the time, a coin will land same way it started. [link]
H = -pi log2 pi
![Page 26: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/26.jpg)
26
![Page 27: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/27.jpg)
27
Are Randall Munroe’s entropy estimates correct?Left for course blog discussion.
![Page 28: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/28.jpg)
28
Class Sorting using “Odette”
Assuming alphabet randomly distributed: O is 15th out of 26 letters in alphabet
0.9880135796356116381290864023063602427958811193759308857790... 0.9880135796356116381290864023063602427958811193759308857790...
= 0.988…
![Page 29: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/29.jpg)
29
Class Sorting using “Odette”
The real class: 41 out of 49 students have names before “Odette”!
0.9880135796356116381290864023063602427958811193759308857790... 0.9880135796356116381290864023063602427958811193759308857790...
-((41/49) * log_2 (41/49) + (7/49) * log_2 (7/49)) = 0.6162235892609252
![Page 30: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/30.jpg)
30
Better Class SortingFront of Class
Hannah – Li-Chang
Anthony - Casey
crm Caroline - Gregory
mdc Michael - Yuan
![Page 31: Class 28: Entropy](https://reader033.fdocuments.net/reader033/viewer/2022061220/54bcbc094a7959410c8b45f0/html5/thumbnails/31.jpg)
Charge
Office hours:Peter – right nowSunday, 1-6pmEveryone should be making progress on PS6
Monday: Trick-or-Treat Protocols!Inheritance