Dynamic Classifier Systems for Classifier Aggregation

Classifier CombiningClassification Confidence

Classifier SystemsExperiments

Assessing Confidence Measures

Dynamic Classifier Systems forClassifier Aggregation

David Stefka

Seminar z umele inteligence11. 12. 2008

Martin Holena

David Stefka Dynamic Classifier Systems for Classifier Aggregation




Outline

1 Classifier Combining

2 Classification Confidence

3 Classifier Systems

4 Experiments

5 Assessing Confidence Measures





Classification

classification – process of assigning patterns into classes

X – feature space, ~x pattern

C1, . . . ,CN – classes

classifier – mapping φ : X → [0, 1]N

φ(~x) = (µ1(~x), . . . , µN(~x))

µi (~x) – degree of classification x ∈ Ci

interpretation of µi (~x) – depends on the classifier used(probability, fuzzy membership, . . . )





Classifier Combining

method for improving the classification by using multipleclassifiers and combining their outputs

create a team T = (φ1, . . . , φr ) of classifiers (bagging,boosting, . . . )

aggregate the team using aggregator Amost of the aggregation methods are static

~x

φ1

φ2

...φr

T (~x) A Φ(~x)





Classification example

C1 – bananas, C2 – apples





Classification example ctnd.

φ(~x) = (0.1, 0.8)






φ(~x) = (0.9, 0.1)






φ(~x) = (0.1, 0.2)

Low confidence





Classification Confidence

degree of trust we can give to the classifier φ

probability of correct classification of ~x by φ

ability to answer ”I don’t know”

confidence measure κφ : X → [0, 1]

static measures – constant of the classifier (e.g., accuracy)

dynamic measures – adapted to the currently classifiedpattern (e.g., local accuracy)





Global Accuracy - GA

static confidence measure

validation (training) set Mproportion of patterns ~y ∈M correctly classified by φ

κ(GA)φ (~x) =

∑~y∈M I (φcr (~y)

?= c(~y))

|M|





Euclidean Local Accuracy - ELA

dynamic confidence measure

validation (training) set MN(x) patterns from M neighboring with ~x (e.g., 20 nearestunder Euclidean metric)

proportion of patterns ~y ∈ N(~x) correctly classified by φ

κ(ELA)φ (~x) =

∑~y∈N(~x) I (φcr (~y)

?= c(~y))

|N(~x)|





Euclidean Local Match - ELM

dynamic confidence measure

validation (training) set MN(x) patterns from M neighboring with ~x (e.g., 20 nearestunder Euclidean metric)

proportion of patterns ~y ∈ N(~x) from the same class as φ ispredicting for ~x

κ(ELM)φ (~x) =

∑~y∈N(~x) I (φcr (~x)

?= c(~y))

|N(~x)|





Classifier Systems

S = (T ,K,A) – classifier system

T = (φ1, . . . , φr ) – classifiers

K = (κφ1 , . . . , κφr ) – confidence measures

A – aggregator

3 types of classifier systems

confidence-freestaticdynamic





Types of classifier systems

~x

φ1

φ2

...φr

T (~x) A Φ(~x)

(a) Confidence-free

~x

φ1

φ2

...φr

T (~x)

Kconst

A Φ(~x)

(b) Static

~x

φ1

φ2

...φr

κφ1

κφ2

...κφr

T (~x)

K(~x)

A Φ(~x)

(c) Dynamic





Mean value based aggregators

confidence-free – mean value of the classifier outputs

static – weighted mean; weights are static confidences

dynamic – weighted mean; weights are dynamic confidences

T (~x) =

0BBB@φ1(~x)φ2(~x)

...φr (~x)

1CCCA =

0BBB@µ1,1(~x) µ1,2(~x) . . . µ1,N(~x)µ2,1(~x) µ2,2(~x) . . . µ2,N(~x)

. . .

µr,1(~x) µr,2(~x) . . . µr,N(~x)

1CCCA , K(~x) =

0BBB@κφ1

(~x)κφ2

(~x)...

κφr (~x)

1CCCA





Experiments

comparison: confidence-free vs. static vs. dynamic systems

systems of Quadratic Discriminant Classifiers

various confidence measures

aggregators based on mean value aggregator

4 artificial, 4 real-world datasets

10-fold crossvalidation

ELM confidence measure





Results – QDC

Confidence−free Static Dynamic0

1

2

3

4

5

6

7

8

Combination method

Me

an

Err

± S

td E

rr

QDC − Breast





Results – QDC


20

21

22

23

24

25

26

27

28

Combination method

Me

an

Err

± S

td E

rrQDC − Clouds





Results – QDC


1.5

2

2.5

3

3.5

4

4.5

5

Combination method

Me

an

Err

± S

td E

rrQDC − Concentric





Results – QDC


19.5

20

20.5

21

21.5

22

22.5

23

23.5

24

Combination method

Me

an

Err

± S

td E

rrQDC − Gauss 3D





Results – QDC


16

18

20

22

24

26

Combination method

Me

an

Err

± S

td E

rrQDC − Phoneme





Results – QDC


18

20

22

24

26

28

30

32

34

Combination method

Me

an

Err

± S

td E

rrQDC − Pima





Results – QDC


13.5

14

14.5

15

15.5

16

16.5

17

17.5

Combination method

Me

an

Err

± S

td E

rrQDC − Satimage





Results – QDC


13

14

15

16

17

18

Combination method

Me

an

Err

± S

td E

rrQDC − Waveform





Experiments - Results

experiments with QDC ensembles: dynamic classifier systemscan significantly improve classification quality

similar results with Random Forests on 17 datasets

improvement in classification quality vs. higher computationalcomplexity

ELM more successfull than ELA






which confidence measure is better?

will a confidence measure bring improvement in theclassification quality?

are the benefits of a dynamic classifier system worth biggercompuational complexity?





OK/NOK Histograms

confidence measure = probability of correct classification?

distribution of correctly classified patterns (OK)

distribution of correctly classified patterns (NOK)

OK and NOK should not overlap and should be separated





OK/NOK Histograms – Waveform

0.0 0.2 0.4 0.6 0.8 1.00

2000

4000

6000

8000

10000ELA

(d) ELA – bad separation

0.0 0.2 0.4 0.6 0.8 1.00

5000

10000

15000

20000

25000

30000ELM

(e) ELM – relatively good separation





OK/NOK Histograms – Phoneme

0.0 0.2 0.4 0.6 0.8 1.00

5000

10000

15000

20000

25000

30000

35000

40000

45000ELA

(a) ELA

0.0 0.2 0.4 0.6 0.8 1.00

10000

20000

30000

40000

50000ELM

(b) ELM





Restricted Histograms

sometimes the OK/NOK histograms do not correlate withimprovement in the prediction

for most of the patterns, the classifiers give similar outputs ⇒the OK/NOK separation is irrelevant

restrict the testing set to “unclear” patterns U(s)

0 ≤ s ≤ r . . . degree of consensus

for any class, at most s classifiers vote for the class

OK/NOK histograms restricted to U(s)





Restricted Histograms – Waveform

0.0 0.2 0.4 0.6 0.8 1.001234567

ELA, s=7

0.0 0.2 0.4 0.6 0.8 1.002468

1012

ELA, s=8

0.0 0.2 0.4 0.6 0.8 1.00

102030405060

ELA, s=9

0.0 0.2 0.4 0.6 0.8 1.00

50100150200250300350400

ELA, s=10

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

10001200

ELA, s=11

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

10001200140016001800

ELA, s=12

0.0 0.2 0.4 0.6 0.8 1.00

5001000150020002500

ELA, s=13

0.0 0.2 0.4 0.6 0.8 1.00

500100015002000250030003500

ELA, s=14

0.0 0.2 0.4 0.6 0.8 1.00

5001000150020002500300035004000

ELA, s=15

0.0 0.2 0.4 0.6 0.8 1.00

10002000300040005000

ELA, s=16

0.0 0.2 0.4 0.6 0.8 1.00

100020003000400050006000

ELA, s=17

0.0 0.2 0.4 0.6 0.8 1.00

1000200030004000500060007000

ELA, s=18

0.0 0.2 0.4 0.6 0.8 1.00

10002000300040005000600070008000

ELA, s=19

0.0 0.2 0.4 0.6 0.8 1.00

2000400060008000

10000ELA, s=20





Restricted Histograms – Waveform

0.0 0.2 0.4 0.6 0.8 1.002468

10121416

ELM, s=7

0.0 0.2 0.4 0.6 0.8 1.00

1020304050

ELM, s=8

0.0 0.2 0.4 0.6 0.8 1.00

20406080

100120140160

ELM, s=9

0.0 0.2 0.4 0.6 0.8 1.00

50100150200250300350

ELM, s=10

0.0 0.2 0.4 0.6 0.8 1.00

100200300400500600700800

ELM, s=11

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

10001200

ELM, s=12

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

1000120014001600

ELM, s=13

0.0 0.2 0.4 0.6 0.8 1.00

500

1000

1500

2000ELM, s=14

0.0 0.2 0.4 0.6 0.8 1.00

50010001500200025003000

ELM, s=15

0.0 0.2 0.4 0.6 0.8 1.00

50010001500200025003000350040004500

ELM, s=16

0.0 0.2 0.4 0.6 0.8 1.00

10002000300040005000600070008000

ELM, s=17

0.0 0.2 0.4 0.6 0.8 1.00

2000400060008000

100001200014000

ELM, s=18

0.0 0.2 0.4 0.6 0.8 1.00

5000

10000

15000

20000ELM, s=19

0.0 0.2 0.4 0.6 0.8 1.00

50001000015000200002500030000

ELM, s=20





Restricted Histograms – Phoneme

0.0 0.2 0.4 0.6 0.8 1.00

50

100

150

200

250ELA, s=10

0.0 0.2 0.4 0.6 0.8 1.00

100200300400500600700

ELA, s=11

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

10001200

ELA, s=12

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

1000120014001600

ELA, s=13

0.0 0.2 0.4 0.6 0.8 1.00

500

1000

1500

2000ELA, s=14

0.0 0.2 0.4 0.6 0.8 1.00

50010001500200025003000

ELA, s=15

0.0 0.2 0.4 0.6 0.8 1.00

500100015002000250030003500

ELA, s=16

0.0 0.2 0.4 0.6 0.8 1.00

5001000150020002500300035004000

ELA, s=17

0.0 0.2 0.4 0.6 0.8 1.00

1000

2000

3000

4000

5000ELA, s=18

0.0 0.2 0.4 0.6 0.8 1.00

10002000300040005000600070008000

ELA, s=19

0.0 0.2 0.4 0.6 0.8 1.00

50001000015000200002500030000350004000045000

ELA, s=20





Restricted Histograms – Phoneme

0.0 0.2 0.4 0.6 0.8 1.00

20406080

100120140160180

ELM, s=10

0.0 0.2 0.4 0.6 0.8 1.00

50100150200250300350400

ELM, s=11

0.0 0.2 0.4 0.6 0.8 1.00

100200300400500600700

ELM, s=12

0.0 0.2 0.4 0.6 0.8 1.00

200

400

600

800

1000ELM, s=13

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

100012001400

ELM, s=14

0.0 0.2 0.4 0.6 0.8 1.00

200400600800

10001200140016001800

ELM, s=15

0.0 0.2 0.4 0.6 0.8 1.00

500

1000

1500

2000

2500ELM, s=16

0.0 0.2 0.4 0.6 0.8 1.00

50010001500200025003000

ELM, s=17

0.0 0.2 0.4 0.6 0.8 1.00

5001000150020002500300035004000

ELM, s=18

0.0 0.2 0.4 0.6 0.8 1.00

2000

4000

6000

8000

10000ELM, s=19

0.0 0.2 0.4 0.6 0.8 1.00

10000

20000

30000

40000

50000ELM, s=20





ROC analysis

convert the histograms to a ROC curve

summary – area under ROC (AUC)

AUC = 50% – random guessing

AUC = 100% – ideal





OK/NOK ROCs – Waveform

0.0 0.2 0.4 0.6 0.8 1.00.0

0.2

0.4

0.6

0.8

1.0ELA

s=7, AUC=0.50

s=8, AUC=0.49

s=9, AUC=0.51

s=10, AUC=0.50

s=11, AUC=0.51

s=12, AUC=0.51

s=13, AUC=0.52

s=14, AUC=0.53

s=15, AUC=0.55

s=16, AUC=0.56

s=17, AUC=0.58

s=18, AUC=0.61

s=19, AUC=0.64

s=20, AUC=0.67

(a) ELA

0.0 0.2 0.4 0.6 0.8 1.00.0

0.2

0.4

0.6

0.8

1.0ELM

s=7, AUC=1.00

s=8, AUC=0.94

s=9, AUC=0.97

s=10, AUC=0.78

s=11, AUC=0.76

s=12, AUC=0.77

s=13, AUC=0.79

s=14, AUC=0.80

s=15, AUC=0.82

s=16, AUC=0.84

s=17, AUC=0.87

s=18, AUC=0.89

s=19, AUC=0.90

s=20, AUC=0.91

(b) ELM





OK/NOK ROCs – Phoneme

0.0 0.2 0.4 0.6 0.8 1.00.0

0.2

0.4

0.6

0.8

1.0ELA

s=10, AUC=0.52

s=11, AUC=0.52

s=12, AUC=0.52

s=13, AUC=0.52

s=14, AUC=0.53

s=15, AUC=0.53

s=16, AUC=0.54

s=17, AUC=0.55

s=18, AUC=0.58

s=19, AUC=0.62

s=20, AUC=0.77

(a) ELA

0.0 0.2 0.4 0.6 0.8 1.00.0

0.2

0.4

0.6

0.8

1.0ELM

s=10, AUC=0.40

s=11, AUC=0.52

s=12, AUC=0.52

s=13, AUC=0.54

s=14, AUC=0.57

s=15, AUC=0.60

s=16, AUC=0.63

s=17, AUC=0.66

s=18, AUC=0.71

s=19, AUC=0.76

s=20, AUC=0.85

(b) ELM





Improvement vs. AUC

AUC for U(s) covering 5% of the data

improvement of DWM over MV

better AUC?⇒ better improvement?

scatterplot for 17 datasets





Improvement vs. AUC

50 60 70 80 90 100AUC [%]

20

0

20

40

60

80

Impro

vem

ent

[%]

ELAELA fitELMELM fit





Conclusions & Future Work

formalism of classifier systems with classification confidence

3 types of classifier systems

experiments with QDCs and RFs: dynamic systems canoutperform both static and confidence-free systems

methods for assessing confidence measures

future work: different classifier types (SVM)

future work: more advanced aggregation methods

better methods for assessing confidence measures





Thank you for your attention

David Stefka

Dynamic Classifier Systemsfor Classifier Aggregation


Dynamic Classifier Systems for Classifier Aggregation

Documents

Transcript of Dynamic Classifier Systems for Classifier Aggregation