ID3 Presentation

8/18/2019 ID3 Presentation

1/28

ID3 Algorithm

Michael Crawford


2/28

Overview

ID3 Background Entropy Shannon Entropy Information ain ID3 Algorithm ID3 E!ample Clo"ing #ote"


3/28

ID3 Background

$Iterative Dichotomi%er 3&'

Invented (y )o"" *uinlan in +,-,'

enerate" Deci"ion .ree" u"ing Shannon

Entropy'

Succeeded (y *uinlan/" C0'1 and C1'2

algorithm"'


4/28

Entropy

In thermodynamic" entropy i" a mea"ureof how ordered or di"ordered a "y"tem i"'

In information theory entropy i" amea"ure of how certain or uncertain thevalue of a random varia(le i" 4or will (e5'

6arying degree" of randomne""depending on the num(er of po""i(levalue" and the total "i%e of the "et'


5/28

Shannon Entropy

Introduced (y Claude Shannon in +,07

*uantifie" $randomne""&

8ower value implie" le"" uncertainty 9igher value implie" more uncertainty


6/28

Information ain

:"e" Shannon Entropy I calculate" effective change in entropy

after making a deci"ion (a"ed on thevalue of an attri(ute'

;or deci"ion tree" it/" ideal to (a"e

deci"ion" on the attri(ute that provide" thelarge"t change in entropy the attri(utewith the highe"t gain'


7/28

Information ain


8/28


9/28

ID3 Algorithm

+5 E"ta(li"h Cla""ification Attri(ute 4in .a(le )5

1 until all attri(ute" have (een u"ed or the "ame

cla""ification value remain" for all row" in the reduced ta(le'


10/28

E!ample


11/28

E!ample

Model Attri(ute can (e to""ed out "inceit" alway" uni?ue and it doe"n/t help our

re"ult'


12/28

E!ample

E"ta(li"h a target cla""ification I" the car fa"t@ =+1 ye" ,+1 no


13/28

E!ample Cla""ification Entropy

Calculating for the Cla""ification Entropy

IE= -(6/15)log2(6/15)-(9/15)log2(9/15) = ~0.971

Mu"t calculate Information ain of remainingattri(ute" to determine the root node'


14/28

E!ample Information ain

Engine = "mall 1 medium 0 large 3 value" for attri(ute engine "o we need 3 entropy calculation"

"mall 1 no + ye" Ismall = -(5/6)log2(5/6)-(1/6)log2(1/6) =

~0.65

medium 3 no < ye" Imedium = -(3/5)log2(3/5)-(2/5)log2(2/5) =~0.97

large < no < ye" Ilarge = 1 (evenly disri!ued su!se)

IGEngine

= IE(S) – [(6/15)*Ismall

+ (5/15)*Imedium

+ (4/15)*Ilarge

]

IEngine 0.971 – 0.85 = 0.121


15/28


SC.ur(o 0 ye" ++ no < value" for attri(ute SC.ur(o "o we need < entropy calculation"

ye" < ye" < no Iur!o = 1 (evenly disri!ued su!se)

no 3 ye" 7 no Inour!o = -(3/11)log2(3/11)-("/11)log2("/11) =~0."#

IGturb

= IE(S) – [(4/15)*Iturb

+ (11/15)*Inturb

]

Itur(o 0.971 – 0.886 = 0.085


16/28


eight = Average 0 8ight 1 9eavy 3 value" for attri(ute weight "o we need 3 entropy calculation"

average 3 no 3 ye" Iaverage = 1 (evenly disri!ued su!se)

light 3 no + ye" Ilig$ = -(3/#)log2(3/#)-(1/#)log2(1/#) =~0."1

heavy 0 no + ye" I$eavy = -(#/5)log2(#/5)-(1/5)log2(1/5) =~0.72

IG!eig"t

= IE(S) – [(6/15)*Ia#erage

+ (4/15)*Ilig"t

+ (5/15)*I"ea#$

]

Ieight 0.971 – 0.856 = 0.115


17/28


;uel Economy < good 3 average +2 (ad 3 value" for attri(ute ;uel Eco "o we need 3 entropy calculation"

good 2 ye" < no Igood = 0 (no varia!iliy)

average 2 ye" 3 no Iaverage = 0 (no varia!iliy)

(ad 1 ye" 1 no I!ad = 1 (evenly disri!ued su!se)

!e %an mit %al%ulatins &r gd and a#erage sin%e t"e$ al'a$send u nt &ast

IGuelE% = IE(S) – [(1/15)*Ibad]

I;uelEco 0.971 – 0.667 = 0.304


18/28

E!ample Choo"ing the )oot #ode

)ecap

IGEngine 1,1

IGturb -5

IG!eig"t 115

IGuelE%

.4

%ur !es &i' is uel *'o+ and ,e 'an immediaely &redi'$e 'ar is no as ,$en uel e'onomy is good or average.


19/28

E!ample )oot of Deci"ion .ree


20/28


21/28


Engine + "mall 1 medium 0 large 3 value" for attri(ute engine "o we need 3 entropy calculation"

"mall + ye" 2 no Ismall = 0 (no varia!iliy)

medium < ye" 3 no Imedium = -(2/5)log2(2/5)-(3/5)log2(3/5) =~0.97

large < no < ye" Ilarge = 1 (evenly disri!ued su!se)

IGEngine

= IE(SuelE%

) – (5/1)*Imedium

+ (4/1)*Ilarge

]

IEngine 1 – 0.885 = 0.115


22/28


SC.ur(o 3 ye" - no < value" for attri(ute SC.ur(o "o we need < entropy calculation"

ye" < ye" + no Iur!o = -(2/3)log2(2/3)-(1/3)log2(1/3) = ~0."#

no 3 ye" 0 no Inour!o = -(3/7)log2(3/7)-(#/7)log2(#/7) = ~0."#

IGturb

= IE(SuelE%

) – [(./1)*Iturb

+ (/1)*Inturb

]

Itur(o 1 – 0.965 = 0.035


23/28


eight 3 average 1 heavy < light 3 value" for attri(ute weight "o we need 3 entropy calculation"

average 3 ye" 2 no Iaverage = 0 (no varia!iliy)

heavy + ye" 0 no I$eavy = -(1/5)log2(1/5)-(#/5)log2(#/5) =~0.72

light + ye" + no Ilig$ = 1 (evenly disri!ued su!se)

IGEngine = IE(Suel E%) – [(5/1)*I"ea#$+(,/1)*Ilig"t]

IEngine 1 – 0.561 = 0.439


24/28

E!ample Choo"ing the 8evel < #ode

)ecap

IGEngine 115

IGturb .5

IG!eig"t 4.0

eig$ $as $e $ig$es gain+ and is $us $e !es '$oi'e.


25/28

E!ample Deci"ion .ree

in'e $ere are only ,oiems or /ur!o ,$ere

eig$ = ig$+ and $eresul is 'onsisen+ ,e 'ansim&liy $e ,eig$ = ig$&a$.


26/28

E!ample :pdated .a(le

ll 'ars ,i$ large engines in $is a!le are no as.

4ue o in'onsisen &aerns in $e daa+ $ere isno ,ay o &ro'eed sin'e medium sie engines maylead o ei$er as or no as.


27/28

Clo"ing #ote"

ID3 attempt" to make the "horte"t

deci"ion tree out of a "et of learning data

"horte"t i" not alway" the (e"tcla""ification'

)e?uire" learning data to have completelycon"i"tent pattern" with no uncertainty'


28/28

)eference"

*uinlan F' ) 4+,715' Induction of Deci"ion .ree" Machine 8earning + 7+>+2= +,7='

)o"" Geter 4+232,--'

Shannon Claude E' Grediction and Entropy of Grinted Engli"h' 4)etrieved

20

ID3 Presentation

Documents

Transcript of ID3 Presentation