Prof. James A. Landay University of Washington Autumn 2008 Heuristic Evaluation October 28, 2008.
Dr. Wei Wei, Prof. James Landay Apr 22, 2020
Transcript of Dr. Wei Wei, Prof. James Landay Apr 22, 2020
![Page 1: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/1.jpg)
Feature Interaction for Interpretability
Apr 22, 2020Dr. Wei Wei, Prof. James Landay
CS 335: Fair, Accountable, and Transparent (FAccT) Deep LearningStanford University
![Page 2: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/2.jpg)
Announcements● Project Proposal Due Apr 24● Mid-way Presentation, May 13
![Page 3: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/3.jpg)
Recap● LIME
○ Optimizes Local Surrogate Loss between predictor f and explanation g
![Page 4: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/4.jpg)
Recap● Anchors are sets of feature predicates applied to the feature space
○ Optimize both Coverage and Precision
![Page 5: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/5.jpg)
RecapLecture 3 Intrinsic Methods for Interpretability
Post Hoc Methods for Interpretability
Lecture 4 Proxy Methods
Lecture 5Feature Interaction
Lecture 6Example Based
Methods
Lecture 7Visualization Based
Methods
![Page 6: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/6.jpg)
Outline● Feature Interaction for Model Interpretability● Layerwise Relevance Propagation● DeepLift● Shapley Additive Explanations (SHAP)
○ Coaliational Game and Shapley Values○ Kernel SHAP○ Deep SHAP○ Tree SHAP
● Equatable Value of Data
![Page 7: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/7.jpg)
Feature Interaction
sample x predictor
z
z
z
z
z
z
zfeature contribution
to predictions
z
z
z
z
f(x)
f(x)
f(x)
f(x)
f(x)
![Page 8: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/8.jpg)
Feature Interaction● Assign Importance Scores to Features
○ Each Feature i in the model will get a value Φi ○ Values explain how ML models make
decisions
![Page 9: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/9.jpg)
Feature Interaction
![Page 10: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/10.jpg)
Outline● Feature Interaction for Model Interpretability● Layerwise Relevance Propagation● DeepLift● Shapley Additive Explanations (SHAP)
○ Coaliational Game and Shapley Values○ Kernel SHAP○ Deep SHAP○ Tree SHAP
● Equatable Value of Data
![Page 11: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/11.jpg)
Layerwise Relevance Propagation (LRP)
Bach et al, 2015
![Page 12: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/12.jpg)
Layerwise Relevance Propagation (LRP)
Relevance Score R Bach et al, 2015
![Page 13: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/13.jpg)
Relevance Scores● xi - output of neuron i● g - activation function● wij - weight of neural network connecting neuron xi and xj● zij - linearly transformed neuron outputs
● Relevant Score Ri(l) of neuron i at level l
![Page 14: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/14.jpg)
Relevance Score Propagation
Bach et al, 2015
![Page 15: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/15.jpg)
Results on Synthetic Data
Bach et al, 2015
![Page 16: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/16.jpg)
Resutls on Pascal Dataset
Bach et al, 2015
![Page 17: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/17.jpg)
More Examples
Bach et al, 2015
![Page 18: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/18.jpg)
Outline● Feature Interaction for Model Interpretability● Layerwise Relevance Propagation● DeepLift● Shapley Additive Explanations (SHAP)
○ Coaliational Game and Shapley Values○ Kernel SHAP○ Deep SHAP○ Tree SHAP
● Equatable Value of Data
![Page 19: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/19.jpg)
● DeepLift Allows Each Neuron A Reference Value for Activation Output xi0
DeepLift
Shrikumar et al, 2016
![Page 21: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/21.jpg)
Results with DNA Pattern Dataset
DeepLift
LRP
Shrikumar et al, 2016
![Page 22: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/22.jpg)
Outline● Feature Interaction for Model Interpretability● Layerwise Relevance Propagation● DeepLift● Shapley Additive Explanations (SHAP)
○ Coaliational Game and Shapley Values○ Kernel SHAP○ Deep SHAP○ Tree SHAP
● Equatable Value of Data
![Page 23: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/23.jpg)
Shapley Additive Explanations (SHAP)● Assigns Feature Importance Weights Based on Game Theory
○ Each feature ia a player○ Probability P(Ŷ | X) is the total payoff○ Distribute the total payoff to players (features) "fairly"
0.6 0.05 0.03 0.01 0.01
0.1 0.2 0.3 -0.1 0
0.2 0.1 0.1 0.2 -0.1
0.05 0.10 0.05 0.1 -0.1
0.7
0.4
0.5
0.2
ΣΦi = P(Ŷ | X)
P(Ŷ | X)Φ0 Φ1 Φ2 Φ3 Φ4
![Page 24: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/24.jpg)
Shapley Values● Developed by Lloyd Shapley
○ American mathematician ○ Nobel Prize-winning economist
![Page 25: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/25.jpg)
Coalitional Game
Players Coalitions Payoffs
v= 150
v= 230
![Page 26: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/26.jpg)
Coalitional Game
Players Coalitions Payoffs
v= -20
v= 230
![Page 27: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/27.jpg)
Coalitional Game● How Do We Assign Importance Scores to Players?
○ Consider the interactions to all other players
How do we account for the interactions among players?
?
? ? ?
v= 100
How much value should we attribute to each player?
![Page 28: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/28.jpg)
Shapley Values● Design A Value Scheme Φi for player i
○ M-Player coalitional game ○ Payoff function v(S)
● Value Scheme Has to Follow Four Criteria○ 1) Efficiency
○ 2) Symmetry
○ 3) Dummy Player
○ 4) Linearity Aas et al, 2019
![Page 29: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/29.jpg)
Shapley Values● Solution: Shparly Values
○ Unique solution that satisfies 1) - 4)○ M - set of players○ v (S) payoff function
impacts to to payoffs
Shapley Value for player i
normalizer
![Page 30: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/30.jpg)
Shapley Values● Calculate Shapley Value for Player 2 1
?
2
?
2
?
231
?
23
![Page 32: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/32.jpg)
Back to ML Interpretability● SHAP
○ Treat each feature i as a player as if we were in a coalitional game○ Estimate the value of feature i by shapley values
Lundberg et al, 2017value of feature i
payoffs (probability)
predictor that uses feature
![Page 33: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/33.jpg)
Additive Feature Attribution
Lundberg et al, 2017
Efficiency of Shparly Values
feature mask
![Page 34: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/34.jpg)
Computational Challenges● Terms Grow In the Order of 2F
● Approximating Solutions○ Shapley Sampling Values (Štrumbelj et al, 2013)○ Tree SHAP (Lundberg et al, 2018)○ Deep Approximate Shapley Propagation (Ancona et al, 2019)
![Page 35: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/35.jpg)
Computational Challenges● Estimating Prediction Outcomes With Partial Features
○ Neural networks are not designed to use partial features○ One solution is to use the expected value (Lundberg et al, 2018)
![Page 36: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/36.jpg)
SHAP Based Methods
SHAP
Kernel SHAP
Equivalent to LIME
Deep SHAP
Equivalent to DeepLift, LRP
Tree SHAP
model agnostic tree based models
linear composition m
odel
![Page 37: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/37.jpg)
Kernel SHAP● Remember the LIME Training Objective
● SHAP Equivalent Objective
Lundberg et al, 2017
![Page 38: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/38.jpg)
Deep SHAP● Incorporate Shapley Values Into Linear Composition Model● DeepLift approximates Deep SHAP when the reference value is taken to be
Lundberg et al, 2017
![Page 39: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/39.jpg)
Feature Importance for MNIST
![Page 40: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/40.jpg)
Outline● Feature Interaction for Model Interpretability● Layerwise Relevance Propagation● DeepLift● Shapley Additive Explanations (SHAP)
○ Coaliational Game and Shapley Values○ Kernel SHAP○ Deep SHAP○ Tree SHAP
● Equatable Value of Data
![Page 41: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/41.jpg)
Tree SHAP● Incorporate Shapley Values to Tree Based Algorithms
○ Implemented into XGBoost and LightGBM○ Reduced the complexity of estimating Shapley Values from O(TL2M ) to O(TLD2 )
■ T - number of trees, L - maximum number of leaves■ M - number of features, D - depth of tree
Lundberg et al, 2018
![Page 42: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/42.jpg)
SHAP Interaction Values● Pairwise Interactions of Shapley Values
Shapley Values
![Page 43: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/43.jpg)
Adult Income● Samples are clustered using the ordering in the leaf nodes
Lundberg et al, 2018
![Page 44: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/44.jpg)
Mortality Data● Survival model on 20 year mortality followup data
○ 14,407 individuals and 79 features
Lundberg et al, 2019
![Page 45: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/45.jpg)
Mortality Data
Lundberg et al, 2019
Shapley Values Shapley Interaction Values
![Page 46: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/46.jpg)
Chronic Kidney Disease
Lundberg et al, 2019
Shapley Values Shapley Interaction Values
![Page 47: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/47.jpg)
Predicting Hypoxaemia During A Surgery● Hypoxaemia
○ An abnormally low amount of oxygen in the blood
Lundberg et al, 2018
![Page 48: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/48.jpg)
Outline● Feature Interaction for Model Interpretability● Layerwise Relevance Propagation● DeepLift● Shapley Additive Explanations (SHAP)
○ Coaliational Game and Shapley Values○ Kernel SHAP○ Deep SHAP○ Tree SHAP
● Equatable Value of Data
![Page 49: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/49.jpg)
Equatable Value of Data Points● Assign Shapley Values to Data Point for A Given Predictor
○ Each data point xi receives a Shapley Value ϕi○ V is a black-box predictor○ C - a constant
Ghorbani et al, 2019
![Page 50: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/50.jpg)
Differentiating Noisy Data Using Shapley Values
Ghorbani et al, 2019
![Page 51: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/51.jpg)
Differentiating Mislabeled Data Using Shapley Values
Ghorbani et al, 2019
![Page 53: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/53.jpg)
Skin Pigmented Lesion Detection● Search Online for Skin Pigmented Lesion Data Using Keyword Search● Use Shapley Values to highlight high value data points
○ Performance improvements 29.6% -> 37.8%
![Page 54: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/54.jpg)
Summary● Feature Interaction
○ An importance score to explain how ML models make decisions○ There exist interactions of features in the same ML model
● SHAP○ Use Shapley Values as Feature Interaction Scores
■ Decomposes model prediction probabilities into an additive model
■ Generalizes LIME, LRP and DeepLift■ Tree SHAP implements SHAP efficiently into tree models
● Data Valuation○ Estimate the value of data points based on Shapley Values
![Page 55: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/55.jpg)
Required Reading● Molnar: Ch 5.9, Ch 5.10
![Page 56: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/56.jpg)
Reading Assignments● Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo
Li, Ce Zhang, Dawn Song, Costas J. Spanos, Towards Efficient Data Valuation Based on the Shapley Value, ICML 2019
● S Chang, Y Zhang, M Yu, T Jaakkola, A Game Theoretic Approach to Class-wise Selective Rationalization, NeurIPS 2019
● Schwab, Patrick, Djordje Miladinovic, and Walter Karlen. Granger-causal attentive mixtures of experts: Learning important features with neural networks, AAAI 2019
● Ying, Zhitao, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. GNNExplainer: Generating explanations for graph neural networks, NeurIPS 2019
● Ancona, Marco, Cengiz Öztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley values approximation, ICML 2019
![Page 57: Dr. Wei Wei, Prof. James Landay Apr 22, 2020](https://reader033.fdocuments.net/reader033/viewer/2022051915/628516412d5c2b6ab11efd2d/html5/thumbnails/57.jpg)
Next Lecture
Example Based Methods for Interpretability