Differential Network Entropy Reveals Cancer System Hallmarks
1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed...
-
Upload
raymond-ryan -
Category
Documents
-
view
215 -
download
2
Transcript of 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed...
![Page 1: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/1.jpg)
1
Lecture 7
System Models
Attributes of a man-made system.
Concerns in the design of a distributed system
Communication channels
Entropy and mutual information
![Page 2: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/2.jpg)
2
Student presentation next week
Up to 5 minute presentations followed by discussions.
All presentations in Power Point
Format: Title of project/research Motivation (why is the problem important) Background (who did what) Specific objectives (what do you plan to do) Literature
Each student will provide feedback about each presentation (grades - A, B, C, F- and comments).
![Page 3: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/3.jpg)
3
Distributed system models
Process A Communication Channel
send(message)
receive(message)
Process B
![Page 4: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/4.jpg)
4
System models
Functional models
Performance models
Reliability models
Security models
The effect of the technology substrate.
![Page 5: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/5.jpg)
5
Attributes of a man-made system
A. Functionality
B Performance and dependabilityReliabilityAvailabilityMaintainabilitySafety
C. Cost
![Page 6: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/6.jpg)
6
Major concerns
Unreliable communication.
Independent failures of communication links and computing nodes.
Discrepancy among communication and computing bandwidth and latency.BandwidthLatency
![Page 7: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/7.jpg)
7
Information transmission and communication channel models
Physical signals
Digital/analog channels
Modulation/demodulation
Sampling and quantization
Channel latency and bandwidth
![Page 8: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/8.jpg)
8
Source B DestinationCommunication Channel
D
propagation delay
transmission time
t1
t2
t3 D/V
L/B
L/B
D/V
L
messagelatency
time
t4
![Page 9: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/9.jpg)
9
EntropyInput and output channel alphabets.
The output of a communication channel depends statistically upon its input. The output gives an idea of what was sent.
Measure of the uncertainty of a random variable.
Examples: Binary random variable
H(x) = -p log(p) – (1-p) log(1-p) Horse race
![Page 10: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/10.jpg)
10
Entropy of a binary random variable
H(X)
p0 1/2 1
1
![Page 11: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/11.jpg)
11
Joint entropy, conditional entropy, mutual information
H(X,Y) – joint entropy of X and YH(X/Y) – conditional entropy of X given YH(X,Y) = H(X) + H(Y/X) = H(Y) + H(X/Y)I(X;Y) = H(X) – H(X/Y) mutual informationI(X;Y) is a measure of the dependency between rv’s X and Y.
H(X) = H(X/Y) + I(X;Y) H(Y) = H(Y/X) + I(X;Y) H(X,Y) = H(X) + H(Y) – I(X;Y)
![Page 12: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/12.jpg)
12
H(X|Y)
H(Y,X)
H(Y|X)
H(X)H(Y)
I(X;Y)
![Page 13: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/13.jpg)
13
Noiseless and noisy binary symmetric channels
x=0 y=0
x=1
Source DestinationNoiseless, binary symmetric communication channel
y=1
x=0 y=0
x=1
SourceDestinationNoisy, binary symmetric communication channel
y=1
(a)
(b)
p1-p
1-p
p
![Page 14: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/14.jpg)
14
Noisy binary symmetric channel
Each of the two input symbols 0 and 1 is altered with probability p and received as 1 and 0 respectively.
Then
I(X;Y) = H(Y) + H(Y/X) =
= H(Y) + p log(p) + (1-p) log(1-p)
We can maximize I(X;Y) when H(Y) = 1
![Page 15: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/15.jpg)
15
Encoding
Encoding used to:Make transmission resilient to errors (error
detection and error correction)Reduce the amount of information transmitted
through a communication channel (compression)Ensure information confidentiality (encryption)
Source Encoding
Channel Encoding
![Page 16: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/16.jpg)
16
Source ReceiverBinary Communication
ChannelSource
Encoder
Source
ChannelDecoder
ChannelEncoder
SourceEncoder
SourceDecoder
A 00B 10C 01D 11
00 A10 B01 C11 D
Binary CommunicationChannel
A 00B 10C 01D 11
SourceDecoder
Receiver00 A10 B01 C11 D
00 0000010 1011001 0101111 11101
00000 0010110 1001011 0111101 11
(a)
(b)
![Page 17: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/17.jpg)
17
Channel capacity and Shannon’s theorem
Given a channel with input X and output Y the channel capacity defines the highest rate the information can be transmitted through the channel
C = max I(X;Y)
Shannon’s theorem
The effect of the signal to noise ratio (S/N)
C = B log ( 1 + S/N)
![Page 18: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/18.jpg)
18
Error detection and error correction
Error detection parity bit used to detect any odd # of errors
Error correction
Code: a set of code words
Block codes:m – information symbolsk – parity check symbolsn = m + k
![Page 19: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/19.jpg)
19
Source Receiver
Communication channelChannelDecoder
ChannelEncoder
original message original message
SourceEncoder
channel encoded message(n-tuples)
Decoder
souce encoded message(k-tuples)
channel encoded message(n-tuples)
souce encoded message(k-tuples)
![Page 20: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/20.jpg)
20
Hamming distance
The number of position two binary code words differ.
Hamming distance is a metricNon-negativeSymmetricTriangle inequality
Example
The distance of a code
Nearest neighbor decoding
![Page 21: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/21.jpg)
21
Error correction and error detection capabilities of a code
If C is an [n,M] code with an odd distance
d = 2e +1
Then C can:
correct e errors and
detect 2e+1 errors
![Page 22: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/22.jpg)
22
c1 r
2e
d(c1,c2) > 2e
d(c3,c4 ) = 2e+1
cux
e e
c2
c3 c4
q
2e
t
![Page 23: 1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.](https://reader035.fdocuments.net/reader035/viewer/2022062805/5697bffb1a28abf838cc0ba2/html5/thumbnails/23.jpg)
23
The Hamming bound
What is the minimum number of parity check symbols necessary to correct one error?