Lecture 22-cs648
-
Upload
rajiv-omar -
Category
Technology
-
view
41 -
download
0
Transcript of Lecture 22-cs648
![Page 1: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/1.jpg)
Randomized AlgorithmsCS648
Lecture 22• Chebyshev Inequality• Method of Bounded Difference
1
![Page 2: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/2.jpg)
Chernoff Bound
Theorem : Suppose be independent Bernoulli random variables with parameters , that is, takes value 1 with probability and 0 with probability . Let and . For any ,
Limitations: • Works only for bounding sum of random variables.• Requires independence among .
![Page 3: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/3.jpg)
THREE EXAMPLES TO ILLUSTRATE THE INAPPLICABILITY OF CHERNOFF BOUND
3
![Page 4: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/4.jpg)
Red-blue balls out of bin
Randomized Experiment:There are red and blue balls in a bag. We take out balls from the bag uniformly randomly and without replacement.
: no. of red balls in the sample.
Aim: To show is concentrated around .
Question: Can we apply Chernoff bound ?Answer: NO because ’s are NOT independent.
4
¿𝒏𝟐
![Page 5: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/5.jpg)
Balls into Bins(number of empty bins)
: random variable denoting the number of empty bins.
[] Aim: To show that is concentrated around .
Question: Can we apply Chernoff bound ?Answer: NO because ’s are NOT independent.
5
1 2 3 … … n
1 2 3 4 5 … m-1 m
¿𝑛(1− 1𝑛 )𝑚
for
![Page 6: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/6.jpg)
Number of Triangles in a random graph
: A graph on vertices where each edge is present with probability independent of others.
: random variable denoting the number of triangles.
[] Aim: To show that is concentrated around .
Question: Can we apply Chernoff bound ?Answer: NO because ’s are NOT independent.
6
𝒂
𝒃𝒄
𝒅
𝒑
for
![Page 7: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/7.jpg)
CHEBYSHEV’S INEQUALITY
7
![Page 8: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/8.jpg)
Chebyshev’s inequality
Let be a random variable defined over a probability space.Question: How to capture deviation of from ?
Define
Question: What is ?Answer:
Redefine
Question: What is ?Answer:
8
Called variance of
![Page 9: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/9.jpg)
Chebyshev’s inequalityLet be a random variable defined over a probability space.
Applying Markov Inequality,
Limitations: • Calculating is sometimes difficult.• Usually gives bounds that are better than Markov Inequality but inferior to the
bound achieved by other methods.• Simple practice problems will be given to you on the use of Chebyshev Inequality.
9
![Page 10: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/10.jpg)
METHOD OF BOUNDED DIFFERENCE (MOBD)
The most powerful method for bounding the probability of deviation of a random variable from expected value
10
![Page 11: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/11.jpg)
The Power of MOBD
• Tightest bound for Randomized Quick Sort was derived using MOBD. : number of comparisons during randomized quick sort on elements.
• MOBD subsumes Chernoff bound.
• Based on theory of Martingales.
Note: Proof similar and almost as hard as the proof of Chernoff bound. [Not part of the course]
11
![Page 12: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/12.jpg)
Notations
: a sequence of random variables.
be a function of random variables.
Objective: to achieve a bound on the probability of deviation of from . = ?
Notations: “ ” means “”
12
![Page 13: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/13.jpg)
A new perspectiveThe value of is well defined once the values taken by is exposed.
View the process of exposing the values of happening gradually in steps.• In the beginning, when none of the is revealed, all we can say about value of is
that its expected value is .• In first step, is exposed. If takes value , all we can say about value of is that its
expected value is .• In second step, is also exposed. If takes value , all we can say about value of is
that its expected value is .
13
…
The next slide will give a visual description of the process mentioned above.
But ponder over this slide before pressing the next button.
![Page 14: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/14.jpg)
The value of and the gradual exposition of ’s
Examples to illustrate the meaning of Algorithm : Quick sort: no. of comparisons during quick sort on elements. : Given first pivot elements, the expected number of comparisons during quick sort on elements.
14
… … … …
𝒙𝟏=𝒂𝟏 𝒙𝟐=𝒂𝟐 𝒙𝟑=𝒂𝟑
…𝒙𝒏=𝒂𝒏
𝐄 [ 𝒇 ∨𝑿¿¿𝟏=𝑨𝟏]¿
𝐄 [ 𝒇 ∨𝑿¿¿𝟐=𝑨𝟐]¿
𝐄 [ 𝒇 ∨𝑿¿¿𝟑=𝑨𝟑]¿ 𝒇 (𝑨𝒏)
𝐄 [ 𝒇 ∨𝑿¿¿𝒏−𝟏=𝑨𝒏−𝟏]¿
𝐄 [ 𝒇 ]
![Page 15: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/15.jpg)
The value of and the gradual exposition of ’s
Examples to illustrate the meaning of Stochastic Process : Ball-Bin problem: no. of empty bins: Given the destination of first balls, the expected number of empty bins.
15
… … … …
𝒙𝟏=𝒂𝟏 𝒙𝟐=𝒂𝟐 𝒙𝟑=𝒂𝟑
…𝒙𝒏=𝒂𝒏
𝐄 [ 𝒇 ∨𝑿¿¿𝟏=𝑨𝟏]¿
𝐄 [ 𝒇 ∨𝑿¿¿𝟐=𝑨𝟐]¿
𝐄 [ 𝒇 ∨𝑿¿¿𝟑=𝑨𝟑]¿ 𝒇 (𝑨𝒏)
𝐄 [ 𝒇 ∨𝑿¿¿𝒏−𝟏=𝑨𝒏−𝟏]¿
𝐄 [ 𝒇 ]
![Page 16: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/16.jpg)
Gradual exposition of ’s
Examples to illustrate the meaning of Random Structure : Random graph: no. of triangles in : Given the presence/absence of edges, the expected number of triangles in the random graph.
16
… … … …
𝒙𝟏=𝒂𝟏 𝒙𝟐=𝒂𝟐 𝒙𝟑=𝒂𝟑
…𝒙𝒏=𝒂𝒏
𝐄 [ 𝒇 ∨𝑿¿¿𝟏=𝑨𝟏]¿
𝐄 [ 𝒇 ∨𝑿¿¿𝟐=𝑨𝟐]¿
𝐄 [ 𝒇 ∨𝑿¿¿𝟑=𝑨𝟑]¿ 𝒇 (𝑨𝒏)
𝐄 [ 𝒇 ∨𝑿¿¿𝒏−𝟏=𝑨𝒏−𝟏]¿
𝐄 [ 𝒇 ]
![Page 17: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/17.jpg)
THE INTUITION UNDERLYING MOBD
17
![Page 18: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/18.jpg)
Method of Bounded Difference
18
…
𝒙𝟏=𝒂𝟏
…
…
𝒙 𝒊+𝟏=𝜶
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊]¿
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊 , 𝒙 𝒊+𝟏=𝜶 ]¿
𝒙 𝒊+𝟏=𝜷
…
𝒙𝟐=𝒂𝟐
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊 , 𝒙 𝒊+𝟏=𝜷 ]¿
𝒇 (𝑨¿¿𝒏)¿
…
𝐄 [ 𝒇 ]
![Page 19: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/19.jpg)
Method of Bounded Difference
For any , and any , , if if is small, thenMost probably will be close to .
Think for a while over the above statement before proceeding further.19
…
𝒙𝟏=𝒂𝟏
…
…
𝒙 𝒊+𝟏=𝜶
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊]¿
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊 , 𝒙 𝒊+𝟏=𝜶 ]¿
𝒙 𝒊+𝟏=𝜷
…
𝒙𝟐=𝒂𝟐
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊 , 𝒙 𝒊+𝟏=𝜷 ]¿
𝐄 [ 𝒇 ]𝒇 (𝑨¿¿𝒏)¿
…
![Page 20: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/20.jpg)
Method of Bounded Difference - I
Theorem 1:If there are positive numbers ’s such that for any , and any , ,
Then
Note: In order to get a meaningful bound using Therem1, you must have small ’s .
20
…
𝒙𝟏=𝒂𝟏
…
…
𝒙 𝒊+𝟏=𝜶
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊]¿
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊 , 𝒙 𝒊+𝟏=𝜶 ]¿
𝒙 𝒊+𝟏=𝜷
…
𝒙𝟐=𝒂𝟐
𝐄 [ 𝒇 ∨𝑿¿¿ 𝒊=𝑨𝒊 , 𝒙 𝒊+𝟏=𝜷 ]¿
𝐄 [ 𝒇 ]𝒇 (𝑨¿¿𝒏)¿
…
![Page 21: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/21.jpg)
Method of Bounded Difference - II
Definition: function is said to satisfy Lipschitz condition with parameters ’s if
For all , that differ only at th coordinate.
Theorem 2: If satisfies Lipschitz condition and are independent, then
Remark: This form is easiest to use but requires independence.
21
![Page 22: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/22.jpg)
MOBD SUBSUMES CHERNOFF BOUND
22
![Page 23: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/23.jpg)
MOBD subsumes Chernoff Bound
are 0-1 independent random variables. satisfies Lipschitz condition with parameters
Hence applying Theorem 2
23
¿𝐞𝐱𝐩 (−𝟐𝒕𝟐
𝒏 ) for
for
![Page 24: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/24.jpg)
PROBLEM 1NO. OF EMPTY BINS
24
![Page 25: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/25.jpg)
Balls into Bins(number of empty bins)
: random variable denoting the number of balls in th bin. : number of empty bins.Observation: takes value in range [, ].
Question: What is s.t. for any given , ?Answer: For , ,
25
1 2 3 … … n
1 2 3 4 5 … n-1 n
𝒏−𝟐−(𝒏−𝟏)(𝟏− 𝟏𝒏−𝟏 )
𝒏
≈𝒏(𝒆−𝟏𝒆 )This will give very inferior bound
![Page 26: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/26.jpg)
Balls into Bins(number of empty bins)
: random variable denoting the destination of th ball. : number of empty bins.Observation:• are independent.• satisfies Lipschitz condition with parameters Hence
26
1 2 3 … … n
1 2 3 4 5 … n-1 n
¿𝐞𝐱𝐩 (−𝟐𝒕𝟐
𝒏 ) for
We failed because our choice of random variables for defining was bad. Can you think of other random
variables such that is a function of them ?
![Page 27: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/27.jpg)
Balls into Bins(number of empty bins)
Theorem: If balls are thrown r.u.i. into bins, then the number of empty bins will be within range with probability at least .
Lesson to be learnt from this exercise: Be careful in selecting the base random variables used in defining .
27
![Page 28: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/28.jpg)
PROBLEM 2RED-BLUE BALLS OUT OF BIN
28
![Page 29: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/29.jpg)
Red-blue balls out of bin
Randomized Experiment:There are red and blue balls in a bag. We take out balls from the bag uniformly randomly and without replacement.
: no. of red balls in the sample.
Observation: are not independent Can apply Theorem 1 only.Question: What is s.t. for any given, ?
29
¿𝒏𝟐
![Page 30: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/30.jpg)
Red-blue balls out of bin
Let has red balls.
Applying Theorem 1
30
…
𝑨𝒊−𝟏 th ball
𝒌+𝟏+(𝒏−𝒌−𝟏𝟐𝒏− 𝒊 )(𝒏− 𝒊)𝒌+( 𝒏−𝒌𝟐𝒏− 𝒊 )(𝒏−𝒊)
¿𝐞𝐱𝐩 (−𝟐𝒕𝟐
𝒏 )
…
…
balls
for
![Page 31: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/31.jpg)
Red-blue balls out of bin
Theorem: There are red and blue balls in a bag. Suppose we take out balls from the bag uniformly randomly and without replacement. The number of red balls in the sample is within range with probability at least .
31
![Page 32: Lecture 22-cs648](https://reader034.fdocuments.net/reader034/viewer/2022052412/558c7887d8b42a646c8b4760/html5/thumbnails/32.jpg)
PROBLEM 3NO. OF TRIANGLES IN RANDOM GRAPH
Do it as exercise.This problem will also be posted in practice sheet.
32