Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize...
Transcript of Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize...
![Page 1: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/1.jpg)
Ensemble Learning and the Heritage Health Prize
Jonathan Stroud, Igii Enverga, Tiffany Silverstein,Brian Song, and Taylor Rogers
iCAMP 2012University of California, Irvine
Advisors: Max Welling, Alexander Ihler, Sungjin Ahn, and Qiang Liu
August 14, 2012
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 2: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/2.jpg)
The Heritage Health Prize
I Goal: Identify patients who will be admitted to a hospitalwithin the next year, using historical claims data.[1]
I 1,250 teams
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 3: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/3.jpg)
Purpose
I Reduce cost of unnecessary hospital admissions per yearI Identify at-risk patients earlier
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 4: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/4.jpg)
Kaggle
I Public online competitions
I Gives feedback on prediction models
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 5: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/5.jpg)
Data
I Provided through Kaggle
I Three years of patient data
I Two years include days spent in hospital (training set)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 6: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/6.jpg)
Evaluation
Root Mean Squared Logarithmic Error (RMSLE)
ε =
√√√√1
n
n∑i
[log(pi + 1) − log(ai + 1)]2
Threshold: ε ≤ .4
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 7: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/7.jpg)
The Netflix Prize
I $1 Million prize
I Leading teams combined predictors to pass threshold
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 8: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/8.jpg)
Blending
Blend several predictors to create a more accurate predictor
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 9: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/9.jpg)
Prediction Models
I Optimized Constant Value
I K-Nearest Neighbors
I Logistic Regression
I Support Vector Regression
I Random Forests
I Gradient Boosting Machines
I Neural Networks
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 10: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/10.jpg)
Feature Selection
I Used Market Makers method [2]
I Reduced each patient to vector of 139 features
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 11: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/11.jpg)
Optimized Constant Value
I Predicts same number of days for each patient
I Best constant prediction is p = 0.209179
RMSLE: 0.486459(800th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 12: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/12.jpg)
K-Nearest Neighbors
I Weighted average of closest neighbors
I Very slow
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 13: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/13.jpg)
Eigenvalue Decomposition
Reduces number of features for each patient
Xk = λ−1/2k UT
k Xc
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 14: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/14.jpg)
K-Nearest Neighbors Results
Neighbors: k = 1000RMSLE: 0.475197
(600th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 15: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/15.jpg)
Logistic Regression
RMSLE: 0.466726(375th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 16: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/16.jpg)
Support Vector Regression
ε = .02RMSLE: 0.467152
(400th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 17: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/17.jpg)
Decision Trees
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 18: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/18.jpg)
Random Forests
RMSLE: 0.464918(315th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 19: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/19.jpg)
Gradient Boosting Machines
Trees = 8000Shrinkage = 0.002
Depth = 7Minimum Observations = 100
RMSLE: 0.462998(200th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 20: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/20.jpg)
Artificial Neural Networks
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 21: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/21.jpg)
Back Propagation in Neural Networking
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 22: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/22.jpg)
Neural Networking Results
Number of hidden neurons = 7Number of cycles = 3000
RMSLE: 0.465705(340th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 23: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/23.jpg)
Individual Predictors (Summary)
I Optimized Constant Value 0.486459 (800th place)
I K-Nearest Neighbors 0.475197 (600th place)
I Logistic Regression 0.466726 (375th place)
I Support Vector Regression 0.467152 (400th place)
I Random Forests 0.464918 (315th place)
I Gradient Boosting Machines 0.462998 (200th place)
I Neural Networks 0.465705 (340th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 24: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/24.jpg)
Individual Predictors (Summary)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 25: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/25.jpg)
Deriving the Blending Algorithm
Error (RMSE)
ε =
√√√√1
n
n∑i=1
(Xi − Yi )2
nε2c =n∑
i=1
(Xi − Yi )2
nε20 =n∑
i=1
Y 2i
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 26: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/26.jpg)
Deriving the Blending Algorithm (Continued)
X as a combination of predictors
X̃ = Xw
or
X̃i =∑c
wcXic
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 27: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/27.jpg)
Deriving the Blending Algorithm (Continued)
Minimizing the cost function
C =1
n
N∑i=1
(Yi − X̃i )2
∂C
∂w=
∑i
(Yi −∑c
wcXic)(−Xic) = 0
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 28: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/28.jpg)
Deriving the Blending Algorithm (Continued)
Minimizing the cost function (continued)∑i
YiXic =∑i
∑c
wcXicXic
Y TX = wTc XT
c X
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 29: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/29.jpg)
Deriving the Blending Algorithm (Continued)
Optimizing predictors’ weights
wc = (Y TX )(XTX )−1∑i
YiXic =∑i
X 2ic +
∑i
Y 2ic −
∑i
(Yi − Xic)2
∑i
YiXic =∑i
X 2ic + nε20 − nε2c
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 30: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/30.jpg)
Deriving the Blending Algorithm (Continued)
Error (RMSE)
ε =
√√√√1
n
n∑i=1
(Xi − Yi )2
nε2c =n∑
i=1
(Xi − Yi )2
nε20 =n∑
i=1
Y 2i
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 31: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/31.jpg)
Deriving the Blending Algorithm (Continued)
Optimizing predictors’ weights
wc = (Y TX )(XTX )−1∑i
YiXic =∑i
X 2ic +
∑i
Y 2ic −
∑i
(Yi − Xic)2
∑i
YiXic =∑i
X 2ic + nε20 − nε2c
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 32: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/32.jpg)
Deriving the Blending Algorithm (Continued)
X as a combination of predictors
X̃ = Xw
or
X̃i =∑c
wcXic
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 33: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/33.jpg)
Blending Algorithm (Summary)
1. Submit and record all predictions X and errors ε
2. Calculate M = (XTX )−1 and
vc = (XTY )c =1
2
∑i (X
2ic + nε20 − nε2c)
3. Because wc = (Y TX )(XTX )−1, calculate weights w = Mv
4. Final blended prediction is X̃i = Xw
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 34: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/34.jpg)
Blending Results
RMSLE: 0.461432(98th place)
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 35: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/35.jpg)
Future Work
I Optimizing Blending Equation with Regularization Constant
wc = (Y TX )(XTX + λI )−1
I Improved feature selection
I More predictors
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 36: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/36.jpg)
Questions
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning
![Page 37: Ensemble Learning and the Heritage Health Prize...Ensemble Learning and the Heritage Health Prize Jonathan Stroud, Igii Enverga, Ti any Silverstein, Brian Song, and Taylor Rogers iCAMP](https://reader034.fdocuments.net/reader034/viewer/2022050413/5f8996009d366f3056027336/html5/thumbnails/37.jpg)
References
Heritage provider network health prize, 2012.http://www.heritagehealthprize.com/c/hhp.
David Vogel Phil Brierley and Randy Axelrod.Market makers - milestone 1 description.September 2011.
Stroud, Enverga, Silverstein, Song, and Rogers Ensemble Learning