Oblique Decision Trees Using Householder Reflection
description
Transcript of Oblique Decision Trees Using Householder Reflection
![Page 1: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/1.jpg)
Oblique Decision Trees Using Householder Reflection
Chitraka Wickramarachchi
Dr. Blair RobertsonDr. Marco RealeDr. Chris PriceProf. Jennifer Brown
![Page 2: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/2.jpg)
Introduction
Literature Review
Methodology
Results and Discussion
Outline of the Presentation
![Page 3: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/3.jpg)
Example: A bank wants to predict the potential status (Default or not) of a new credit card customer
For the existing customers the bank has following data
Introduction
Salary No of Credit cards
Total Credit Amount
No of Loans
Total Loan Installment
Value of Other earnings
Gender Age
Possible approach - Generalized Linear models with binomial errors
Model become complex if the structure of the data is complex.
![Page 4: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/4.jpg)
Decision tree is a tree structured classifier.
Decision Tree (DT)
Salary <= s
TCA < tc
TLA < tl
DND D
Root Node
Non-Terminal
Node
Terminal Node
Test based on features
![Page 5: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/5.jpg)
Recursively partition the feature space into disjoint sub-
regions until each sub-region becomes homogeneous with
respect to a particular class
Partitions
X1
X2
![Page 6: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/6.jpg)
Choosing the best split
0.0221
0.0895 0.154
6
0.1123
0.00150.154
6
0.0654
0.1586
0.1412
0.1224
0.0345
0.1586
X2 <= 0.6819
X1<= 0.4026X1<= 0.5713
![Page 7: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/7.jpg)
Types of DTs
Decision Trees
Univariate DT
Multivariate DT
Linear DTNon-Linear
DT
Axis parallel splits
Oblique splits
![Page 8: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/8.jpg)
Axis parallel splits
Easy to implement Computer complexity is low Easy to interpret
Advantages
Disadvantage
When the true boundaries are
not axis parallel it produces
complicated boundary structure
![Page 9: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/9.jpg)
Axis parallel boundaries
X1
X2
![Page 10: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/10.jpg)
Oblique splits
Advantage - Simple boundary structure
X1
X2
![Page 11: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/11.jpg)
Disadvantages
Implementation is challenging Computer complexity is high
Therefore computationally less expensive oblique tree
induction method would be desirable
X1
X2
Oblique splits
![Page 12: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/12.jpg)
Literature Review
Oblique splits search for splits in the form of
CART – LC Starts with the best axis parallel split Perturb each coefficient until find the best split
Breiman et al. (1984)
Can get trapped in local mimina
Limitations
No upper bound on the time spent at any node
∑𝑖=1
𝑑
𝑎𝑖 𝑥 𝑖+𝑎0≤𝑐
![Page 13: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/13.jpg)
Literature Review
Heath et al. (1993)
Simulated annealing Decision Trees (SADT)
First places a hyperplane in a canonical location Perturb each coefficient randomly
By randomization - try to escape from the local mimima
Algorithm runs much slower than CART- LC
Limitations
![Page 14: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/14.jpg)
Literature Review
Murthy et al. (1994)
Oblique Classifier 1 (OC1)
Start with the best axis parallel split Perturb each coefficient At a local mimima, perturb the hyperlane randomly
Since 1994, there are many ODT induction methods have been
developed based on EA algorithms and neural network
concept
![Page 15: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/15.jpg)
Proposed Methodology
Our approach is to
Transform the data set parallel to one of the
feature axes
Implement axis parallel splits
Back-transform them in to the original space
Transformation is done using Householder reflection.
![Page 16: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/16.jpg)
Householder Reflection
Let X and Y are vectors with the same norm there exists
orthogonal symmetric matrix P such that
where𝒀=𝑯𝑿 𝑯= 𝑰−𝟐𝑾𝑾𝑻 𝒂𝒏𝒅𝑾=𝑿 −𝒀
‖𝑿−𝒀‖𝟐
![Page 17: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/17.jpg)
Orientation of a cluster can be represented by the dominant
Eigen vector of its variance covariance matrix.
X1
X2
Householder Reflection
![Page 18: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/18.jpg)
Householder Reflection
𝒆𝟏=𝑯𝒅
𝑯= 𝑰−𝟐𝑾𝑾𝑻 𝒂𝒏𝒅𝑾=𝒅−𝒆𝟏
‖𝒅−𝒆𝟏‖𝟐
![Page 19: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/19.jpg)
X1
X2
Householder Reflection
![Page 20: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/20.jpg)
To avoid over-fitting
Number of Terminal Nodes
Accuracy
Cost-complexity pruning
![Page 21: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/21.jpg)
Results and Discussion
Data sets - UCI Machine Learning Repository
Data set Number of examples
Number of features
Number of Classes
Iris Data 150 4 3
Boston Housing Data 506 13 2
Estimate of the accuracy was obtained by ten 5-fold
cross validation experiments.
![Page 22: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/22.jpg)
Results and Discussion
Classifer Iris Data Housing Data
Householder Method
CART-LC
OC1
C4.5
Results High accuracy Computationally inexpensive
![Page 23: Oblique Decision Trees Using Householder Reflection](https://reader038.fdocuments.net/reader038/viewer/2022110105/56816979550346895de16f92/html5/thumbnails/23.jpg)
THANK YOU