MIS2502: Data Analytics Classification using Decision Trees

MIS2502:Data AnalyticsClassification using Decision Trees

What is classification?

• Determining to what group a data element belongs– Or “attributes” of that “entity”

• Examples– Determining whether a customer

should be given a loan– Flagging a credit card transaction

as a fraudulent charge– Categorizing a news story as

finance, entertainment, or sports

How classification works

•Choose a set of records for the training set1•Choose a set of records for the validation set2•Choose a class attribute (classifier) 3•Find model that predicts the class attribute as a function of the other attributes4•Apply that model to the validation set to check accuracy5•Apply the final model to future records to classify6

Decision Tree LearningTrans.ID

Charge Amount

Avg. Charge 6 months

Item Same state as billing

Classification

1 $800 $100 Electronics No Fraudulent

2 $60 $100 Gas Yes Legitimate

3 $1 $50 Gas No Fraudulent

4 $200 $100 Restaurant Yes Legitimate

5 $50 $40 Gas No Legitimate

6 $80 $80 Groceries Yes Legitimate

7 $140 $100 Retail No Legitimate

8 $140 $100 Retail No Fraudulent

Trans.ID

Charge Amount

Avg. Charge 6 months

Item Same state as billing

Classification

101 $100 $200 Electronics Yes ?

102 $200 $100 Groceries No ?

103 $1 $100 Gas Yes ?

104 $30 $25 Restaurant Yes ?

Training Set

Validation Set

Derive model

Apply model

Classification software

Goals

The trained model should assign new cases to the right category

It won’t be 100% accurate, but should be as close as possible

The model’s rules can be applied to new records as they come along

An automated, reliable way to predict the outcome

Classification Method: The Decision Tree

• A model to predict membership of cases or values of a dependent variable based on one or more predictor variables (Tan, Steinback, and Kumar 2004)

Example: Credit Card Default

TID Income Debt Owns/Rents

Outcome

1 25k 35% Owns Default

2 35k 40% Rent Default

3 33k 15% Owns No default

4 28k 19% Rents Default




8 85k 10% Rents No default

…

PredictorsClassification

Training Data

We create the tree from a set of training data

Each unique combination of predictors is associated with an outcome

This set was “rigged” so that every combination is accounted for and has an outcome

Example: Credit Card Default

Credi tApproval

Inc ome<40k

Debt > 20%

Owns house

Default

Rents

Default

Debt < 20%

Owns house

No D efault

Rents

Default

Inc ome>40k

Debt > 20%

Owns house

No D efault

Rents

Default

Debt < 20%

Owns house

No D efault

Rents

No D efault

Root node

Child nodeLeafnode


Outcome









…


Training Data

Same Data, Different Tree

We just changed the order of the predictors.


Outcome









…


Training Data

Apply to new (validation) data


Decision(Predicted)

Decision (Actual)

1 80k 35% Rent Default No Default

2 20k 40% Owns Default Default

3 15k 15% Owns No Default No Default

4 50k 19% Rents No Default Default

5 35k 30% Owns Default No Default

Validation DataCredi t

Approv al

Income<40k

Debt > 20%

Owns house

Default

Rents

Default

Debt < 20%

Owns house

No Default

Rents

Default

Income>40k

Debt > 20%

Owns house

No Default

Rents

Default

Debt < 20%

Owns house

No Default

Rents

No Default

How well did the decision tree do in predicting the

outcome?

When it’s “good enough,” we’ve got our model for future decisions.

In a real situation…

The tree induction software has to deal with instances where…

The same set of predictors resulting in different outcomes

Multiple paths result in the same outcome

Not every combination of predictors is in the training set

Tree Induction Algorithms

CreditApproval

Income<40k

Debt > 20%

Owns houseDefault

RentsDefault

Debt < 20%

Owns houseNo Default

RentsDefault

Income>40k

Debt > 20%Owns houseNo Default

RentsDefault

Debt < 20%Owns house

No Default

Rents

No Default

Tree induction algorithms take large

sets of data and compute the tree

Similar cases may have different

outcomes

So probability of an outcome is

computed

For instance, you may find:When income > 40k, debt < 20%, and the customers rents

no default occurs 80% of the time.

0.8

How the induction algorithm works

Start with single node with all training data

Are samples all of same classification?

Are there predictor(s) that will split the data?

Partition node into child nodes according to predictor(s)

Are there more nodes (i.e., new child nodes)? DONE!

Yes

No

No

Yes

No

Go to next node.

Yes

Start with root node

CreditApproval

CreditApproval

Income<40k

Debt > 20%Owns house

Default

RentsDefault

Debt < 20%Owns houseNo Default

RentsDefault

Income>40k


RentsDefault


RentsNo Default

• There are both “defaults” and “no defaults” in the set

• So we need to look for predictors to split the data

Split on income

CreditApproval

Income <40k

Income >40k

CreditApproval

Income<40k


Default

RentsDefault


RentsDefault

Income>40k


RentsDefault


RentsNo Default

• Income is a factor• (Income < 40, Debt > 20, Owns = Default)

but (Income > 40, Debt > 20, Owns = No default)

• But there are also a combination of defaults and no defaults within each income group

• So look for another split

Split on debt

CreditApproval

Income <40k

Debt >20%

Debt <20%

Income >40k

Debt >20%

Debt <20%

CreditApproval

Income<40k


Default

RentsDefault


RentsDefault

Income>40k


RentsDefault


RentsNo Default

• Debt is a factor• (Income < 40, Debt < 20, Owns = No

default) but (Income < 40, Debt < 20, Rents = Default)

• But there are also a combination of defaults and no defaults within some debt groups

• So look for another split

Split on Owns/Rents

Inc ome <40k

Debt >20%

Defa ult

Debt <20%

Owns house

N o Default

Rents

Defa ult

Inc ome >40k

Debt >20%

Owns house

N o Default

Rents

Defa ult

Debt <20%

N o Default

CreditApproval

Income<40k


Default

RentsDefault


RentsDefault

Income>40k


RentsDefault


RentsNo Default

• Owns/Rents is a factor

• For some cases it doesn’t matter, but for some it does

• So you group similar branches

• …And we stop because we’re out of predictors!

How does it know when and how to split?

There are statistics that show• When a predictor variable

maximizes distinct outcomes(if age is a predictor, we should see that older people buy; younger people don’t)

• When a predictor variable separates outcomes(if age is a predictor, we should not see older people who buy mixed up with older people who don’t)

Example: Chi-squared test

Is the proportion of the outcome class the same in each child node?

It shouldn’t be, or the classification isn’t very helpful

Root (n=1500)Default = 750

No Default = 750

Owns (n=850)Default = 300

No Default = 550

Rents (n=650)Default = 450

No Default = 200

ij

ijij

EEO 2

2

ObservedOwns Rents

Default 300 450 750

No Default 550 200 750

850 650 1500

ExpectedOwns Rents

Default 425 325 750


850 650 1500

Chi-squared test

ij

ijij

EEO 2

2

If the groups were the same, you’d expect an even split (Expected)

But we can see they aren’t distributed evenly (Observed)

But is it enough (i.e., statistically significant)?

325)325200(

325)325450(

425)425550(

425)425300( 2222

2

4.1690.480.487.367.362

0001.0p

Small p-values (i.e., less than 0.05 mean it’s very unlikely the groups are the same)

So Owns/Rents is a predictor that creates two different groups

Observed

Owns Rents

Default 300 450 750


850 650 1500

Expected

Owns Rents

Default 425 325 750


850 650 1500

Bottom line: Interpreting the Chi-Squared Test

• High statistic (low p-value) from chi-squared test means the groups are different

• SAS shows you the logworth value– Which is -log(p-value)– Way to compare split variables (big logworths = ) – Low p-values are better:-log(0.05) 2.99-log(0.0001) 9.21

Reading the SAS Decision TreeOutcome cases are not 100% certain– There are probabilities attached to each outcome in a

node– So let’s code “Default” as 1 and “No Default” as 0

So what is the chance that:

A renter making more than $40,000 and debt

more than 20% of income will default?

A home owner making less than $40,000 and debt more than 20%

of income will default?

MIS2502: Data Analytics Classification using Decision Trees

Documents

Transcript of MIS2502: Data Analytics Classification using Decision Trees