Vapnik-Chervonenkis Dimension

Part I: Definition and Lower bound

PAC Learning model

• There exists a distribution D over domain X• Examples: <x, c(x)>

– use c for target function (rather than ct)

• Goal: – With high probability (1-)– find h in H such that – error(h,c ) < – arbitrarily small.

VC: Motivation

• Handle infinite classes.

• VC-dim “replaces” finite class size.

• Previous lecture (on PAC):– specific examples– rectangle.– interval.

• Goal: develop a general methodology.

Definitions: Projection

• Given a concept c over X– associate it with a set (all positive examples)

• Projection (sets)– For a concept class C and subset S– C(S) = { c S | c C}

• Projection (vectors)– For a concept class C and S = {x1, … , xm}– C(S) = {<c(x1), … , cxm)> | c C}

Definition: VC-dim

• Clearly |C(S) | 2m

• C shatters S if |C(S) | =2m

• VC dimension of a class C:– The size d of the largest set S that shatters C.– Can be infinite.

• For a finite class C– VC-dim(C) log |C|

Example 1: Interval

1 0

C1={cz | z [0,1] }

cz(x) = 1 x z

Example 2: line

C2={cw | w=(a,b,c) }

cw(x,y) = 1 ax+by c

Example 3: Parallel Rectangle

Example 4: Finite union of intervals

Example 5 : Parity

• n Boolean input variables

• T {1, …, n}

• fT(x) = iT xi

• Lower bound: n unit vectors

• Upper bound– Number of concepts– Linear dependency

Example 6: OR

• n Boolean input variables• P and N subsets {1, …, n}• fP,N(x) = ( iP xi) ( iN xi)• Lower bound: n unit vectors• Upper bound

– Trivial 2n– Use ELIM (get n+1)– Show second vector removes 2 (get n)

Example 7: Convex polygons

Example 7: Convex polygons

Example 8: Hyper-plane

• VC-dim(C8) = d+1

• Lower bound– unit vectors and zero vector

• Upper bound!

C8={cw,c | wd}

cw,c(x) = 1 <w,x> c

Radon Theorem

• Definitions:– Convex set.– Convex hull: conv(S)

• Theorem:– Let T be a set of d+2 points in Rd

– There exists a subset S of T such that– conv(S) conv(T \ S)

• Proof!

Hyper-plane: Finishing the proof

• Assume d+2 points T can be shattered.

• Use Radon Theorem to find S such that– conv(S) conv(T \ S)

• Assign point in S label 1– points not in S label 0

• There is a separating hyper-plane

• How will it label conv(S) conv(T \ S)

Lower bounds: Setting

• Static learning algorithm:– asks for a sample S of size m()– Based on S selects a hypothesis

Lower bounds: Setting

• Theorem:– if VC-dim(C) = then C is not learnable.

• Proof:– Let m = m(0.1,0.1)– Find 2m points which are shattered (set T)– Let D be the uniform distribution on T– Set ct(xi)=1 with probability ½.

• Expected error ¼.• Finish proof!

Lower Bound: Feasible

• Theorem– VC-dim(C)=d+1, then m()=(d/)

• Proof:– Let T be a set of d+1 points which is shattered.– D samples:

• z0 with prob. 1-8

• zi with prob. 8/d

Continue

– Set ct(z0)=1 and ct(zi)=1 with probability ½

• Expected error 2• Bound confidence

– for accuracy

Lower Bound: Non-Feasible

• Theorem– For two hypoth. m()=((log 1))

• Proof:– Let H={h0, h1}, where hb(x)=b

– Two distributions:

– D0: Prob. <x,1> is ½ - and <y,0> is ½ +

– D1: Prob. <x,1> is ½ + and <y,0> is ½ -