Shapley Explanation Networks · 2020. 12. 11. · 2.2 Shallow SHAPNET To aid in our notation, we...

Shapley Explanation Networks

Rui Wang, Xiaoqian Wang, David I. InouyeSchool of Electrical and Computer Engineering

Purdue UniversityWest Lafayette, IN 47906

{ruiw, joywang, dinoye}@purdue.edu

Abstract

We propose to incorporate Shapley values themselves as latent representationsin deep models. This intrinsic explanation approach enables (1) layer-wise ex-planations, (2) explanation regularization of the model during training, and (3)fast explanation computation at test time. We define the Shapley transform thattransforms the input into a Shapley representation given some scalar functions. Weoperationalize the Shapley transform as a neural network module and constructboth shallow and deep networks, called SHAPNETs, by composing Shapley mod-ules. We prove that our Shallow SHAPNETs compute the exact Shapley values andour Deep SHAPNETs maintain the missingness and accuracy properties of Shapleyvalues. We showcase on synthetic and real world datasets the three new abilities ofour SHAPNETs while maintaining reasonable classification accuracy.

1 Introduction

While Shapley values have been shown to be the most theoretically-grounded additive featureattribution explanation methods, the exact (or even approximate) computation of Shapley valuesis challenging due to the exponential time complexity. Thus, several approximation methods havebeen proposed [1, 2, 3]. To avoid approximation, the model class could be restricted [4]. However,even if the computational drawback is overcome, the Shapley values cannot aid in model design ortraining. On the other hand, Generalized Additive Models (GAM) can be seen as interpretable modelthat exposes the exact Shapley values directly, where the prediction and the corresponding exactSHAP explanation can be computed simultaneously. However, GAM models are inherently limitedin representational power, particularly for data like images where deep networks are state-of-the-art.We present related works that motivated our work above and in the text where appropriate. Due tospace limit, for an extended literature review, we refer to Appendix F.

To overcome these drawbacks, we propose to incorporate Shapley values themselves as learned latentrepresentations (as opposed to post-hoc) in deep models—thereby making Shapley explanationsfirst-class citizens in the modeling paradigm. Intuitive illustrations of such representation are providedin Fig. 4 in the appendix and detailed discussion in subsection 2.1. We summarize our contributions:

• We formally define the Shapley transform and prove a simple but useful linearity property forconstructing networks. We also develop important extensions of the base case to enhance therepresentational power of the transform.

• We develop a novel network architecture SHAPNET, which include Shallow and Deep SHAPNET,that intrinsically provides layer-wise explanations in the same forward pass as the prediction.

• We prove that a Shallow SHAPNET explanations are the exact Shapley values and prove thatDeep SHAPNET explanations maintain the missingness and local accuracy properties.

• We propose a novel explanation regularization on Shapley values of the model during training.

1st NeurIPS workshop on Interpretable Inductive Biases and Physically Structured Learning (2020), virtual.

• We demonstrate empirically that our SHAPNETs can provide new capabilities such as layer-wiseexplanations and novel explanation regularizations while maintaining comparable performanceto other deep models. We also show that our method compares favorably with Shapley-basedpost-hoc explanation methods.

2 Shapley explanation networks

Background Given a model C : Rd 7→ R, additive feature-attribution methods form a linearapproximation of the function over simplified binary inputs, z ∈ {0, 1}d, indicating the “presence”and “absence”, respectively: i.e., a local linear approximation D(z) = a0 +

∑di=1 aizi. While there

are different ways to model “absence” and “presence”, in this work, “presence” means that we keepthe original value whereas “absence” means we replace the original value with a reference value,validated in [5] as Baseline Shapley. Denote the reference vector for all features by r, then we candefine a mapping between z and x as Ψx,r(z) = z�x+(1−z)�r, where� denotes element-wiseproduct (e.g., Ψx,r([0, 1]) = [r1, x2]). [1] propose three properties that additive feature attributionmethods should satisfy: local accuracy, missingness, and consistency (see Appendix A for details).Definition 1 (SHAP Values [1]). SHAP values are defined as:

φi (C,x) ,∑

z̃⊆z\i

|z̃|! (d− |z̃| − 1)!d!

[C ◦Ψx,r (z̃ ∪ {i})− C ◦Ψx,r (z̃)] , (1)

where |z̃| is the number of non-zero entries, z \ i means zj = 1 for all j 6= i and zi = 0, z̃ ⊆ z \ irepresents all z̃ whose non-zeros are subsets of the non-zero entries of z\i.

2.1 Shapley transform and Shapley representation

We now present various definitions and properties to describe our approach including the Shapleytransform, the Shapley representation, and a Shapley module.

Definition 2 (Shapley transform). Given a c-tuple of scalar functions f (1:c) , [f (1), · · · , f (c)] ∈ Fcwhere F = {f : Rd 7→ R}, we define the Shapley transform Ω: (Rd ×Fc) 7→ Rc×d to be:

Ω(x, f (1:c)) ,

φ(f (1),x)T

...φ(f (c),x)T

=

φ1(f

(1),x) · · · φd(f (1),x)...

. . ....

φ1(f(c),x) · · · φd(f (c),x)

, (2)

where φ(f (j),x) ∈ Rd for 1 ≤ j ≤ c is the Shapley value vector of x with respect to function f (j).Definition 3 (Shapley representation). Given a tuple of functions f (1:c) as in Def. 2 and an inputinstance x, we simply define a Shapley representation Z ∈ Rc×d to be: Z , Ω(x, f (1:c)).Remark 4 (Channel dimension of representation). We will denote the dimension c (rows of therepresentation) the channel dimension as an analogy to the channel dimension in images.Lemma 5 (Linear transformations of Shapley transforms are Shapley transforms). The linear trans-form, denoted by a matrix A ∈ Rc′×c, of a Shapley representation, Z , Ω(x, f (1:c)), is itself aShapley transform for modified functions f̃ (1:c

′), i.e.,

AZ ≡ AΩ(x, f (1:c)) = Ω(x, f̃ (1:c′)) where f̃ (k)(x) = aTk f (1:c)(x) , (3)ak ∈ Rc is the k-th column of A, and f (1:c)(x) , [f (1)(x), · · · , f (c)(x)] ∈ Rc.Definition 6 (Shapley Module). Given an arbitrarily complex function f (j) : Rd 7→ R parameterizedby a neural network, and a reference vector r ∈ Rd, a Shapley module F : Rd 7→ Rd is definedas: F (x | f (j), r) ,

[φ1(f

(j),x), . . . , φd(f(j),x)

]T, where φi(f (j),x) = 0 if i ∈ D(f (j)), where

D(f (j)) = {i : ∀xi, ∂f (j)/∂xi = 0} is the dummy set and its complement, the active set, A(f (j)).

With low cardinality for the active sets (e.g., the function ignores all but 2 variables), we can make thethe computation for Z in Def. 3 tractable and the Shapley representation itself will be row-wise sparse.Examples of a Shapley module with 2 inputs can be seen in Fig. 5 in the appendix. Additionally, wecan extend our Shapley transform to the case of vector-valued functions and tuples of vector valuedfunctions as we detail in Remark 15 & Remark 16 in the appendix.

2

2.2 Shallow SHAPNET

To aid in our notation, we briefly define a simple linear dimension-wise summation operator.Definition 7 (Dimension Sum Operator sum[α]). Given an input tensor X ∈ Rα1×α2×α3···, sum[αj ]will denote an operator that sums along the j-th dimension corresponding to the letter in brackets.

For example, if X ∈ Rc×d, then sum[c](X) = ∑ci=1 xi. We present our Shallow SHAPNET and itsaccompanying Theorem 9 which is an extension of Lemma 5 with complete proof in Appendix D.Definition 8 (Shallow SHAPNET). A Shallow SHAPNET G is defined as

G(x) = sum[d] ◦ g(x; f (1:c)) = sum[d] ◦ (sum[c] ◦ Ω(x, f (1:c))) =d∑

i=1

c∑

j=1

φi(f(j),x), (4)

where g(x; f (1:c)) will be called the Shallow SHAPNET explanation.

Theorem 9. Shallow SHAPNETs compute the exact Shapley values, i.e., gi(x; f (1:c)) = φi(G,x).

An example of a Shallow SHAPNET is in Fig. 5 in the appendix.

2.3 Deep SHAPNET

To enable inter-layer Shapley representations within a deep model, we construct deep SHAPNETs bycascading the output of Shallow SHAPNETs from Def. 8. However, given the extension to tuples ofvector-valued functions (Remark 16 in appendix), the Shapley representation Z ∈ Rn×d is a matrix,and that would mean the next Shallow SHAPNET layer inside the deep SHAPNET would have amatrix as input (as opposed to a feature vector). This is done with the generalized Shapley Transform(denoted as S) detailed in Def. 18 in the appendix.Definition 10 (Deep SHAPNET). A Deep SHAPNETH is defined as a cascade of Shallow SHAPNETswith generalized Shapley transformH:H = sum[d] ◦ h = sum[d] ◦ sum[c] ◦ S(L) ◦ sum[c] ◦ S(L−1) · · · ◦ sum[c] ◦ S(2) ◦ sum[c] ◦ Ω, (5)

where h(x) is known as the Deep SHAPNET explanation and all the reference values are set to 0except for the first Shapley transform Ω (whose reference values will depend on the application).

We present the theoretical properties about Deep SHAPNETs explanations (proof in Appendix E):Theorem 11. The Deep SHAPNET explanations h(x) satisfy both local accuracy and missingness.

2.4 Explanation regularization during training

One of the important benefits to SHAPNETs is that we can regularize the explanations during training.The main idea is to regularize the last layer explanation so that the model learns to attribute featuresin ways aligned with human priors–e.g. sparse or smooth. For `1 regularization, we would like themodel to focus on a small set of features. For `∞ regularization, the idea is to smooth the Shapleyvalues so that none of the input features become too important individually. Finally, with ad-hocknowledge on the attribution for the task at hand, the user could specify specific regularizations.

3 Experiments & Visualizations

With MNIST [6], FashionMNIST [7], and Cifar-10 [8], we (1) validate that our models can be quiteexpressive despite the intrinsic explanation design, (2) demonstrate that our intrinsic SHAPNETexplanations perform better than posthoc explanations, and (3) highlight the novel ability of layer-wise explanations and explanation regularization on Shapley values. Extensive experiments are donein lower dimensional datasets and details provided in Appendix C. More details in Appendix J.

SHAPNET model performance We report classification accuracies on MNIST [6], FashionM-NIST [7], Cifar-10 [8] as 99.50%, 91.95%, and 82.06%, respectively. While there is some gapbetween state-of-the-art for image data, our image-based deep SHAPNET models can indeed do rea-sonably well even on these high-dimensional non-linear classification problems, all while providingfundamentally new abilities as explored later. Details in Appendix J.

3

0 100 200 300 400 500 600 700 800# of features removed

0.0

0.2

0.4

0.6

0.8

1.0

%of

orig

inal

accu

racy

SHAPNET Explanations Deep LIFT Scaled explanations Input×Gradient Integrated Gradients Input Gradient adjusted for signs

0 20 40 60 80 100 120# of features removed

0.0

0.2

0.4

0.6

0.8

1.0

%of

orig

inal

accu

racy

Figure 1: Our intrinsic Deep SHAPNET explanations perform better than post-hoc explanationsin identifying the features that contribute most to the prediction as seen in this figure showing theaccuracy after removing the top k features identified by each explanation method. Figure on the rightis a zoomed-in version to show more detail for the first 120 features.

Intrinsic SHAPNET explanations compared to post-hoc explanations First, because our intrinsicSHAPNET explanations are simultaneously produced with the prediction, the explanation time ismerely the cost of a single forward pass (A wall-clock time experiment in Appendix G). For image-based data, we evaluate our high-dimensional explanations based on dropping the top-k features asidentified by the explanation and measuring the remaining accuracy of the classifier. We compareDeep SHAPNET explanations with DeepLIFT [9] (scaled to get DeepSHAP [1]), Integrated Gradients[10], Input×Gradients[11] and vanilla gradients adjusted for signs. As shown in Fig. 1, DeepSHAPNET offers high-quality explanations as the model’s accuracy dropped the fastest.

New capability: layer-wise explanations With the layer-wise explanation structure, we can probeinto the Shapley representation at each stage of a Deep SHAPNET as in Fig. 2. We also provide a setof pruning experiment shown in Fig. 7 of Appendix I, where we show how pruning the values in theShapley representations of different layer changes model performance.

New capability: Explanation regularization during training For MNIST data, we visualize theexplanations from different regularization models in Fig. 3. For a more rigorous study of regularizerson image datasets, we perform the top-k feature removal experiment with different regularizers inFig. 6, which shows that our regularizers produce some model robustness towards missing features.Moreover, Fig. 7 in Appendix I also shows that `1 regularized model achieves the best result undervalue pruning across different layers and `∞ performs the worst. Details are in Appendix I.

AAAGcHicpVTLbtNQEJ0Gakp5tbBgAQJDVQlVNNgVEl1WsAGJRYsaUqmNkO3ctFb8wr6GVlGWfANb2PAN/RX+AMSKP+DM2GnSJnEr4as447lzztx5XTcJ/Exb1s+Z2qXLs8aVuavz167fuHlrYfH2+yzOU081vDiI0x3XyVTgR6qhfR2onSRVTugGqul2X/F+85NKMz+OtvVRolqhsx/5Hd9zNFStPa0Ode9NlOQ6639YWLLqljzmuGCXwtLG/eOtP18eHG/Gi7UftEdtismjnEJSFJGGHJBDGdYu2WRRAl2LetClkHzZV9SneWBzWClYONB28d7H126pjfDNnJmgPXgJ8EuBNGkZmBh2KWT2Zsp+LsysncbdE04+2xH+3ZIrhFbTAbTn4QaWF8e5sKqOVVOH1iVGHzEnouHovdJLLlnjyMyRqDUYEuhYbmM/hewJclAHUzCZ5IZz78j+L7FkLX97pW1OvyujYJ+BxPE/kbDPQKr4GXJhvwqEOWI/nX8b3cTcHeCjU8zFWhbOTPK0Dl4XrJwFxpknqKooD9AVRU45R50R7qGHoU0gp+5eKG/mlMWMGXhCIIsKN+gdvZX6Fn6YVUsHROCs6rMYqyt94cJi0tkTnCZGXEomwJeMcEVW6SNYHYmI/ZolS9E/583EMLOTfLoyn4ziE3IftqEdTI55Mok83VW+Irk5uCpFJ0/yxXdAWCL1GHfRAzwvNj2jNXoKScl9Va/wG4pfztugIv0xv2dtYmBTmU/uluncPLcs8wkPie9f++xtOy401+r287ptb+EmfknFM0f36DE9QWAvaINe0ybayENRv9I3+j7717hrPDQeFaa1mRJzh049xso/PME7Qg==

InputsAAAGd3icpVTLbtNQEJ0GGkp5pWUJqiyqoi5oaldIdFnBhgWLFDWkUltVtnOTWvEL+xpaRVnyGyyBT+iv8AcgfoAlZ8ZOkzaJWwlfxRnPnXPmzus6se+l2jR/zlVu3Z6v3lm4u3jv/oOHj2pLyx/SKEtc1XQjP0r2HTtVvheqpva0r/bjRNmB46uW03vD+61PKkm9KNzTZ7E6Cuxu6HU819ZQHdeWD7U61f1GEnUTlbLZ4Li2atZNeYxJwSqE1Z2n57t/vqycN6Klyjc6pDZF5FJGASkKSUP2yaYU64AsMimG7oj60CWQPNlXNKBFYDNYKVjY0Pbw7uLroNCG+GbOVNAuvPj4JUAatAZMBLsEMnszZD8TZtbO4u4LJ5/tDP9OwRVAq+kE2utwQ8ub4xxYlceqqUPbEqOHmGPRcPRu4SWTrHFkxljUGgwxdCy3sZ9AdgU5rIMhmFRyw7m3Zf+XWLKWv93CNqPfpVGwT1/i+J9I2KcvVfwMObffAMIYs5/Nv4duYu4O8OEl5nytCWcqedoGrwNWzgLjjAtUWZQn6Io8p5yjzhj3yMPIxpdT926UN2PGYsYUPAGQeYWb9J7eSX1zP8yqpQNCcJb1WYTVk75wYDHt7DFOEyEuJRPgSUa4Ihv0Eay2RMR+jYIl75/rZmKU2Wk+HZlPRvEJuQ/b0A4nx7iYRJ7uMl+h3BxclbyTp/niOyAokHqCO+8BnheLNmmLXkBScl/VS/wG4pfzNqzIYMLvVZsI2ETmk7tlNjfPLct8wlPi+9e6ettOCq2tuvWyblm7uIlfU/4s0BN6RusI7BXt0FtqoI1cUH6l7/Rj/m91pfq8up6bVuYKzGO69FStf7/cPb0=

Progression

Figure 2: Deep SHAPNETs enable us to peek into the progression of feature importance during theforward pass with layer-wise explainability. The blocks appear in early stages presumably becausethe first two layers computes the Shapley representations in a relatively small neighborhood, andsince all the neighboring pixels share the same values, they output the same blocks. We simply takethe average of the contribution over all channels in the Shapley representation and scale to [−1, 1].Expanded figures (Fig. 11, Fig. 12, Fig. 13, & Fig. 14) in subsection J.5 with discussions.

AAAGfnicpVTLbtNQEJ0WGkp5NIUVAiGLKhILmsYVEllWsGHBokUNrdRWke3cpFauH9jX0BJFrPgUWMK2v8IfgPgIODN2mrR5tBK+ijOeO+fMndd1Y+2nplb7OTd/7fpC6cbizaVbt+/cXS6v3HuXRlniqYYX6SjZc51UaT9UDeMbrfbiRDmBq9Wu233F+7sfVJL6UbhjTmJ1GDid0G/7nmOgapYfHCitm7Z1YNSx6SWqk2kn8T+pVr9ZXq1Va/JY44JdCKubj063/3x5fLoVrcx/owNqUUQeZRSQopAMZE0OpVj7ZFONYugOqQddAsmXfUV9WgI2g5WChQNtF+8OvvYLbYhv5kwF7cGLxi8B0qIKMBHsEsjszZL9TJhZO427J5x8thP8uwVXAK2hI2gvww0sr45zYTU7VkNtqkuMPmKORcPRe4WXTLLGkVkjURswxNCx3MJ+AtkT5KAOlmBSyQ3n3pH9X2LJWv72CtuMfs+Mgn1qieN/ImGfWqr4EXJuvwaENWI/nX8H3cTcbeDDc8z5qghnKnmqg9cFK2eBcdYZalaUR+iKPKeco/YI99DD0EbLqbtXyps1ZTFjCp4AyLzCDXpLb6S+uR9mNdIBIThn9VmE1ZW+cGEx6ewxThMhLiUT4EtGuCJr9B6sjkTEfq2CJe+fy2ZimNlJPl2ZT0bxCbkPW9AOJsc6m0Se7lm+Qrk5uCp5J0/yxXdAUCDNGHfeAzwvNq3TBj2DpOS+qs7wG4hfztugIv0xvxdtImATmU/uluncPLcs8wmPie9f++JtOy7sblTt51Xb3sZN/JLyZ5Ee0hN6isBe0Ca9pi20kUef6St9px8Lf0uV0lppPTednysw9+ncU6r/A942QE8=

`1regularizedAAAGg3icpVTLbtNQEJ2WEkp5tbAEVRZVJYRoiKNKdINUwYYFixY1pFJTRbZz01q5fmBfQ0OUHfwKC7aw6K/wByB+gjNjp0mbRyvhqzjjuXPO3HldN9Z+aiqVX3Pz1xaul24s3ly6dfvO3XvLK/ffp1GWeKrmRTpK9l0nVdoPVc34Rqv9OFFO4GpVdzuveb/+USWpH4V7phurw8A5Cv227zkGqubyakNp3Wz4Ydt0rYZRJ6aXqKNMO4n/WbX6zeW1SrkijzUu2IWwtv3odPfv19XTnWhl/hs1qEUReZRRQIpCMpA1OZRiHZBNFYqhO6QedAkkX/YV9WkJ2AxWChYOtB28j/B1UGhDfDNnKmgPXjR+CZAWrQMTwS6BzN4s2c+EmbXTuHvCyWfr4t8tuAJoDR1DexluYHl1nAur2bEaatOWxOgj5lg0HL1XeMkkaxyZNRK1AUMMHcst7CeQPUEO6mAJJpXccO4d2f8tlqzlb6+wzejPzCjYp5Y4/icS9qmlip8g5/YbQFgj9tP599BNzN0GPjzHnK914UwlT1vgdcHKWWCcdYaaFeUxuiLPKeeoPcI99DC00XLqzpXyZk1ZzJiCJwAyr3CN3tFbqW/uh1mNdEAIzll9FmF1pC9cWEw6e4zTRIhLyQT4khGuyAZ9AKsjEbFfq2DJ++eymRhmdpJPV+aTUXxC7sMWtIPJsc4mkad7lq9Qbg6uSt7Jk3zxHRAUSDPGnfcAz4tNz6lKzyApua/KM/wG4pfzNqhIf8zvRZsI2ETmk7tlOjfPLct8whPi+9e+eNuOC/Vq2d4s2/YubuJXlD+L9JAe0xME9oK26Q3toI08+kLf6Qf9LC2Unpaqpc3cdH6uwDygc0/p5T/C30HX

`1regularizedAAAGcXicpVTNThNRFD5UqYh/oBsTjbmBkJgotUNIZEl048IFGGpJAM3M9LZMOn/O3EFI06Xv4FYXPgOv4hto3PkEfufMlBbaDiTOTadnzj3fd+75u07se6mp13/OVK5dn63emLs5f+v2nbv3Fhbvv0+jLHF1w438KNl17FT7XqgbxjO+3o0TbQeOr5tO9zXvN490knpRuGNOYn0Q2J3Qa3uubaD6sG/0seklupP5dtL/uLBcr9XlUeOCVQjLm49Pt/98eXK6FS1WftA+tSgilzIKSFNIBrJPNqVYe2RRnWLoDqgHXQLJk31NfZoHNoOVhoUNbRfvDr72Cm2Ib+ZMBe3Ci49fAqSiFWAi2CWQ2ZuS/UyYWTuNuyecfLYT/DsFVwCtoUNoL8MNLK+Oc2BVHquhNm1IjB5ijkXD0buFl0yyxpGpkagNGGLoWG5hP4HsCnJQByWYVHLDubdl/5dYspa/3cI2o9+lUbBPX+L4n0jYpy9V/Aw5t18FQo3YT+ffQTcxdxv48BxzvlaEM5U8bYDXAStngXHqDFUW5SG6Is8p56g9wj30MLTx5dTdK+VNTVnMmIInADKvcIPe0Vupb+6HWY10QAjOsj6LsLrSFw4sJp09xmkixKVlAjzJCFdklT6B1ZaI2K8qWPL+uWwmhpmd5NOR+WQUn5D7sAXtYHLU2STydJf5CuXm4KrknTzJF98BQYE0Y9x5D/C8WPSC1ug5JC33Va3EbyB+OW+DivTH/F60iYBNZD65W6Zz89yyzCc8Jr5/rYu37bjQXKtZ6zXL2sZN/IryZ44e0RI9RWAvaZPe0BbaiA/0lb7R99m/1YdVVV3KTSszBeYBnXuqz/4BBnc7uw==

regular

Figure 3: MNIST SHAPNET explanations for different regularizations qualitatively demonstrate theeffects of regularization. We notice that `1 only puts importance on a few key features of the digitswhile `∞ spreads out the contribution over more of the image. Red and blue correspond to positiveand negative contributions respectively. More visualization of the explanations, including the otherclasses and more in-depth discussion, can be found in subsection J.4.

4

Broader Impact

In our work, we proposed a complex neural network model that is intrinsically explainable, whichallows for explanation regularization. This answers two questions (1)“Why?”, as people try to figureout why a neural network is making a specific decision, and (2) after an explanation for a complexneural network model is generated, the next natural question would be “What next?”.

Who may benefit from this research

Those deep learning practitioners or users who value explainability and who want to know how toimprove their model based on explanations can largely benefit from this research. This may includepractitioners of critical tasks, as our research provides a way to answer both “why?” and “whatnext?”, which are important questions to ask and answer for tasks with high stakes.

From an ethical point of view, our research allows one to, during training, limit or entirely eliminatethe influence of elements of human social biases like race, gender, or age, further promoting equalityand fairness in machine learning systems.

Who may be put at disadvantage from this research

We note that our work provide a degree of freedom which could help or hurt depending upon whouse it. If the work of this research is deployed, one can argue that people expecting the highestperformance of a system may be disappointed as our research limits the model choices.

Also, one could argue the ability to eliminate certain feature contribution may allow practitionerswith ill-intentions to build models that might escalate the discrimination.

Consequences of failure of the system

In the case of a failure of our system, where we would assume it results in non-ideal approximationto the true Shapley values, as investigated by literature [12], a malicious builder can fool the userby developing a model whose approximation of Shapley values are so off that the underlying modelis actually discriminating against people without the approximation of the explanation showing so.However, since the work in [12] relies on methods who requires out-of-sample inputs, which ours donot, fooling our model can be a bit more intricate and complex.

Whether the task/method leverages biases in the data

Our work allows the users to leverage biases as well as to limit or eliminate those. As discussedpreviously, the impact of such ability largely depends on the intent of the users.

Acknowledgments

R.W. and D.I. acknowledge support from Northrop Grumman Corporation through the NorthropGrumman Cybersecurity Research Consortium (NGCRC) and the Army Research Lab throughcontract number W911NF-2020-221. X.W. acknowledges support from NSF III #1955890.

5

References[1] S. M. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,”

in Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg,S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. CurranAssociates, Inc., 2017, pp. 4765–4774. [Online]. Available: http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf

[2] M. Ancona, C. Oztireli, and M. Gross, “Explaining deep neural networks with a polynomial timealgorithm for shapley value approximation,” in Proceedings of the 36th International Conferenceon Machine Learning, ser. Proceedings of Machine Learning Research, K. Chaudhuri andR. Salakhutdinov, Eds., vol. 97. Long Beach, California, USA: PMLR, 09–15 Jun 2019, pp.272–281. [Online]. Available: http://proceedings.mlr.press/v97/ancona19a.html

[3] J. Chen, L. Song, M. J. Wainwright, and M. I. Jordan, “L-shapley and c-shapley: Efficient modelinterpretation for structured data,” in International Conference on Learning Representations,2019. [Online]. Available: https://openreview.net/forum?id=S1E3Ko09F7

[4] S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb,N. Bansal, and S.-I. Lee, “From local explanations to global understanding with explainable AIfor trees,” Nature Machine Intelligence, vol. 2, no. 1, pp. 56–67, Jan. 2020. [Online]. Available:https://doi.org/10.1038/s42256-019-0138-9

[5] M. Sundararajan and A. Najmi, “The many shapley values for model explanation,” ser.Proceedings of Machine Learning Research, Vienna, Austria, 12–18 July 2020. [Online].Available: http://https://arxiv.org/abs/1908.08474

[6] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010. [Online]. Available:http://yann.lecun.com/exdb/mnist/

[7] H. Xiao, K. Rasul, and R. Vollgraf. (2017) Fashion-mnist: a novel image dataset for benchmark-ing machine learning algorithms.

[8] A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009.[9] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating

activation differences,” in Proceedings of the 34th International Conference on MachineLearning, ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds.,vol. 70. International Convention Centre, Sydney, Australia: PMLR, 06–11 Aug 2017, pp.3145–3153. [Online]. Available: http://proceedings.mlr.press/v70/shrikumar17a.html

[10] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” inProceedings of the 34th International Conference on Machine Learning, ser. Proceedings ofMachine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. International ConventionCentre, Sydney, Australia: PMLR, 06–11 Aug 2017, pp. 3319–3328. [Online]. Available:http://proceedings.mlr.press/v70/sundararajan17a.html

[11] A. Shrikumar, P. Greenside, A. Shcherbina, and A. Kundaje, “Not just a black box: Learningimportant features through propagating activation differences,” CoRR, vol. abs/1605.01713,2016. [Online]. Available: http://arxiv.org/abs/1605.01713

[12] D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju, “Fooling lime and shap: Adversarialattacks on post hoc explanation methods,” in Proceedings of the AAAI/ACM Conference onAI, Ethics, and Society, ser. AIES ’20. New York, NY, USA: Association for ComputingMachinery, 2020, p. 180–186. [Online]. Available: https://doi.org/10.1145/3375627.3375830

[13] G. Desjardins, K. Simonyan, R. Pascanu, and k. kavukcuoglu, “Natural neural networks,” inAdvances in neural information processing systems 28, C. Cortes, N. D. Lawrence, D. D. Lee,M. Sugiyama, and R. Garnett, Eds. Curran Associates, Inc., 2015, pp. 2071–2079. [Online].Available: http://papers.nips.cc/paper/5953-natural-neural-networks.pdf

[14] L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using Real NVP,” ICLR, 2017.[Online]. Available: https://openreview.net/forum?id=HkpbnH9lx

[15] D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1convolutions,” in Advances in Neural Information Processing Systems 31, S. Bengio,H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. CurranAssociates, Inc., 2018, pp. 10 215–10 224. [Online]. Available: http://papers.nips.cc/paper/8224-glow-generative-flow-with-invertible-1x1-convolutions.pdf

6

http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdfhttp://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdfhttp://proceedings.mlr.press/v97/ancona19a.htmlhttps://openreview.net/forum?id=S1E3Ko09F7https://doi.org/10.1038/s42256-019-0138-9http://https://arxiv.org/abs/1908.08474http://yann.lecun.com/exdb/mnist/http://proceedings.mlr.press/v70/shrikumar17a.htmlhttp://proceedings.mlr.press/v70/sundararajan17a.htmlhttp://arxiv.org/abs/1605.01713https://doi.org/10.1145/3375627.3375830http://papers.nips.cc/paper/5953-natural-neural-networks.pdfhttps://openreview.net/forum?id=HkpbnH9lxhttp://papers.nips.cc/paper/8224-glow-generative-flow-with-invertible-1x1-convolutions.pdfhttp://papers.nips.cc/paper/8224-glow-generative-flow-with-invertible-1x1-convolutions.pdf

[16] L. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking atrous convolution forsemantic image segmentation,” CoRR, vol. abs/1706.05587, 2017. [Online]. Available:http://arxiv.org/abs/1706.05587

[17] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic imagesegmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEETransactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, pp. 834–848, 2018.

[18] D. Dua and C. Graff, “UCI machine learning repository,” 2017. [Online]. Available:http://archive.ics.uci.edu/ml

[19] L. S. Shapley, “A value for n-person games,” Contributions to the Theory of Games, vol. 2,no. 28, pp. 307–317, 1953.

[20] M. T. Ribeiro, S. Singh, and C. Guestrin, ““why should i trust you?”: Explaining thepredictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD InternationalConference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY,USA: Association for Computing Machinery, 2016, p. 1135–1144. [Online]. Available:https://doi.org/10.1145/2939672.2939778

[21] ——, “Anchors: High-precision model-agnostic explanations,” in AAAI Conference on ArtificialIntelligence, 2018. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16982

[22] R. Guidotti, A. Monreale, S. Ruggieri, D. Pedreschi, F. Turini, and F. Giannotti, “Localrule-based explanations of black box decision systems,” CoRR, vol. abs/1805.10820, 2018.[Online]. Available: http://arxiv.org/abs/1805.10820

[23] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi, “A survey ofmethods for explaining black box models,” ACM Comput. Surv., vol. 51, no. 5, Aug. 2018.[Online]. Available: https://doi.org/10.1145/3236009

[24] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualisingimage classification models and saliency maps,” in Workshop at International Conference onLearning Representations, 2014.

[25] Y. Lou, R. Caruana, and J. Gehrke, “Intelligible models for classification and regression,” inProceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery andData Mining, ser. KDD ’12. New York, NY, USA: Association for Computing Machinery,2012, p. 150–158. [Online]. Available: https://doi.org/10.1145/2339530.2339556

[26] Y. Lou, R. Caruana, J. Gehrke, and G. Hooker, “Accurate intelligible models with pairwiseinteractions,” in Proceedings of the 19th ACM SIGKDD International Conference on KnowledgeDiscovery and Data Mining, ser. KDD ’13. New York, NY, USA: Association for ComputingMachinery, 2013, p. 623–631. [Online]. Available: https://doi.org/10.1145/2487575.2487579

[27] R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and N. Elhadad, “Intelligible models forhealthcare: Predicting pneumonia risk and hospital 30-day readmission,” in Proceedings of the21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser.KDD ’15. New York, NY, USA: Association for Computing Machinery, 2015, p. 1721–1730.[Online]. Available: https://doi.org/10.1145/2783258.2788613

[28] D. Alvarez Melis and T. Jaakkola, “Towards robust interpretability with self-explainingneural networks,” in Advances in Neural Information Processing Systems 31, S. Bengio,H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. CurranAssociates, Inc., 2018, pp. 7775–7784. [Online]. Available: http://papers.nips.cc/paper/8003-towards-robust-interpretability-with-self-explaining-neural-networks.pdf

[29] A. S. Ross, M. C. Hughes, and F. Doshi-Velez, “Right for the Right Reasons: TrainingDifferentiable Models by Constraining their Explanations,” arXiv:1703.03717 [cs, stat],May 2017, tex.ids: rossRightRightReasons2017a arXiv: 1703.03717. [Online]. Available:http://arxiv.org/abs/1703.03717

[30] F. Liu and B. Avci, “Incorporating priors with feature attribution on text classification,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 6274–6283. [Online].Available: https://www.aclweb.org/anthology/P19-1631

7

http://arxiv.org/abs/1706.05587http://archive.ics.uci.edu/mlhttps://doi.org/10.1145/2939672.2939778https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16982https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16982http://arxiv.org/abs/1805.10820https://doi.org/10.1145/3236009https://doi.org/10.1145/2339530.2339556https://doi.org/10.1145/2487575.2487579https://doi.org/10.1145/2783258.2788613http://papers.nips.cc/paper/8003-towards-robust-interpretability-with-self-explaining-neural-networks.pdfhttp://papers.nips.cc/paper/8003-towards-robust-interpretability-with-self-explaining-neural-networks.pdfhttp://arxiv.org/abs/1703.03717https://www.aclweb.org/anthology/P19-1631

[31] G. G. Erion, J. D. Janizek, P. Sturmfels, S. Lundberg, and S. Lee, “Learning explainablemodels using attribution priors,” CoRR, vol. abs/1906.10670, 2019. [Online]. Available:http://arxiv.org/abs/1906.10670

[32] L. Rieger, C. Singh, W. J. Murdoch, and B. Yu, “Interpretations are useful: penalizingexplanations to align neural networks with prior knowledge,” ser. Proceedings ofMachine Learning Research, Vienna, Austria, 12–18 July 2020. [Online]. Available:http://https://arxiv.org/abs/1909.13584

[33] W. J. Murdoch, P. J. Liu, and B. Yu, “Beyond word importance: Contextual decomposition toextract interactions from LSTMs,” in International Conference on Learning Representations,2018. [Online]. Available: https://openreview.net/forum?id=rkRwGg-0Z

[34] C. Singh, W. J. Murdoch, and B. Yu, “Hierarchical interpretations for neural networkpredictions,” in International Conference on Learning Representations, 2019. [Online].Available: https://openreview.net/forum?id=SkEqro0ctQ

[35] D. Smilkov, N. Thorat, B. Kim, F. B. Viégas, and M. Wattenberg, “Smoothgrad:removing noise by adding noise,” CoRR, vol. abs/1706.03825, 2017. [Online]. Available:http://arxiv.org/abs/1706.03825

[36] C.-K. Yeh, C.-Y. Hsieh, A. Suggala, D. I. Inouye, and P. K. Ravikumar, “On the(in)fidelity and sensitivity of explanations,” in Advances in Neural Information ProcessingSystems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, andR. Garnett, Eds. Curran Associates, Inc., 2019, pp. 10 967–10 978. [Online]. Available:http://papers.nips.cc/paper/9278-on-the-infidelity-and-sensitivity-of-explanations.pdf

[37] M. Ancona, C. Öztireli, and M. Gross, “Shapley Value as Principled Metric for StructuredNetwork Pruning,” arXiv:2006.01795 [cs], Jun. 2020, arXiv: 2006.01795. [Online]. Available:http://arxiv.org/abs/2006.01795

[38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014, citearxiv:1412.6980Comment: Published as a conference paper at the 3rd InternationalConference for Learning Representations, San Diego, 2015. [Online]. Available:http://arxiv.org/abs/1412.6980

[39] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin,N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani,S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperativestyle, high-performance deep learning library,” in Advances in Neural Information ProcessingSystems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett,Eds. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

[40] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” inProceedings of the 27th International Conference on International Conference on MachineLearning, ser. ICML’10. Madison, WI, USA: Omnipress, 2010, p. 807–814.

[41] P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for Activation Functions,”arXiv:1710.05941 [cs], Oct. 2017, arXiv: 1710.05941. [Online]. Available: http://arxiv.org/abs/1710.05941

[42] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” arXiv:1607.06450 [cs, stat], Jul.2016, arXiv: 1607.06450. [Online]. Available: http://arxiv.org/abs/1607.06450

[43] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical Evaluation of Rectified Activations inConvolutional Network,” arXiv:1505.00853 [cs, stat], Nov. 2015, arXiv: 1505.00853. [Online].Available: http://arxiv.org/abs/1505.00853

8

http://arxiv.org/abs/1906.10670http://https://arxiv.org/abs/1909.13584https://openreview.net/forum?id=rkRwGg-0Zhttps://openreview.net/forum?id=SkEqro0ctQhttp://arxiv.org/abs/1706.03825http://papers.nips.cc/paper/9278-on-the-infidelity-and-sensitivity-of-explanations.pdfhttp://arxiv.org/abs/2006.01795http://arxiv.org/abs/1412.6980http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfhttp://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdfhttp://arxiv.org/abs/1710.05941http://arxiv.org/abs/1710.05941http://arxiv.org/abs/1607.06450http://arxiv.org/abs/1505.00853

AAAGhHicpVTLbtNQEJ0GMKW8UliysYgqFZEGO1DRDaiCDQsWBTVtpT4i27lprfhVPyBVlC2fwoot/At/AH/BmbHTpE3iVsJXTefOnXPmzuvakecmqWH8XqjcuHlLu714Z+nuvfsPHlaXH+0kYRY7quWEXhjv2VaiPDdQrdRNPbUXxcrybU/t2r33fL77RcWJGwbb6VmkDn3rOHC7rmOlULWrene13zbrer/dfPbmIDmN0wH2R83nUBw1h2tmu1ozGoZ8+rRgFkKNim8rXK58pwPqUEgOZeSTooBSyB5ZlGDtk0kGRdAd0gC6GJIr54qGtARsBisFCwvaHn6PsdsvtAH2zJkI2oEXD38xkDqtABPCLobM3nQ5z4SZtfO4B8LJdzvDf7vg8qFN6QTaq3Ajy+vjbFiVx5pSlzYkRhcxR6Lh6J3CSyZZ48j0iahTMETQsdzBeQzZEeSoDrpgEskN596S8z9iyVreO4VtRn9Lo2CfnsTxP5GwT0+q+BVybr8GhD5hP59/G93E3F3ggwvM+VoRzkTytAFeG6ycBcbp56iyKE/QFXlOOUfdCe6xh7GNJ7fuXStv+pzFjAl4fCDzCrfoM32U+uZ+mDWVDgjAWdZnIVZP+sKGxay7R7hNiLiUTIArGeGKrNEpWC2JiP3qBUveP1fNxDizs3zaMp+M4htyH3agHU2Ofj6JPN1lvgJ5ObgqeSfP8sVvgF8g0ynuvAd4Xkx6QU2qQ1LyXjVK/Pril/M2qshwyu9lmxDYWOaTu2U+N88ty3zDPg3x/pqXX9tpYafZMNcbxqdXtc13xUu8SE/oKa0irte0SR9oC13k0Df6QT/pl6Zpde2ltp6bVhYKzGO68Glv/wFXaTvj

f(x1, x2) =q

x21 + x22 � 1

AAAGg3icpVTLbtNQEJ0Gakp5pbBkgUVUqYg22FEQ3VSqYMOCRUFNW6mpItu5aa34hR+QKsqSX2HDFj6GP4C/4MzYadImcSvhqzjjuXPO3HldO/LcJDWM30uVW7eXtTsrd1fv3X/w8FF17fFBEmaxo1pO6IXxkW0lynMD1Urd1FNHUaws3/bUod1/x/uHX1ScuGGwn55H6sS3TgO35zpWClWn+qy3MeiYm/qg03ixA6mdur5K2okbDJvQjTrVmlE35NFnBbMQalQ8e+Fa5Tu1qUshOZSRT4oCSiF7ZFGCdUwmGRRBd0JD6GJIruwrGtEqsBmsFCwsaPt4n+LruNAG+GbORNAOvHj4xUDqtA5MCLsYMnvTZT8TZtYu4h4KJ5/tHP92weVDm9IZtNfhxpY3x9mwKo81pR5tS4wuYo5Ew9E7hZdMssaR6VNRp2CIoGO5i/0YsiPIcR10wSSSG869Jft/xJK1/O0Uthn9LY2CfXoSx/9Ewj49qeJXyLn9FhD6lP1i/n10E3P3gA8uMedrXTgTydM2eG2wchYYp1+gyqI8Q1fkOeUc9aa4Jx4mNp6cun+jvOkLFjMm4PGBzCvcok/0Qeqb+2HWVDogAGdZn4VYfekLGxbzzh7hNCHiUjIBrmSEK7JFn8FqSUTsVy9Y8v65biYmmZ3n05b5ZBSfkPuwC+14cvSLSeTpLvMVyM3BVck7eZ4vvgP8ApnOcOc9wPNi0itq0CYkJfdVvcSvL345b+OKjGb8XrUJgY1lPrlbFnPz3LLMJxwQ37/m1dt2Vjho1M3XdeNjs7b7triJV+gpPacNxPWGduk97aGLHPpGP+gn/dKWtZdaQ2vmppWlAvOELj3azj9aejxi

f(x1, x2) = x1 ⇥ sin 4x2

AAAGZXicpVTNbtNAEJ4Gakqh0ALiwgGLqBIHGuwIRI8VXDhwKNC0lUpV2c6mtbL+wV5DqyqPwBXegHfiCYC34Jux06RN4lbCqzjj2fm+2flbP9Vhbhzn11zj2vV568bCzcVbt5fu3F1eubedJ0UWqE6Q6CTb9b1c6TBWHRMarXbTTHmRr9WO33/D+ztfVJaHSbxlTlK1H3mHcdgLA89A9fH4oH2w3HRajjz2pOBWQpOqZzNZafykT9SlhAIqKCJFMRnImjzKsfbIJYdS6PbpFLoMUij7iga0CGwBKwULD9o+3of42qu0Mb6ZMxd0AC8avwxIm1aBSWCXQWZvtuwXwszaWdynwslnO8G/X3FF0Bo6gvYy3NDy6jgfVvWxGurRusQYIuZUNBx9UHkpJGscmT0WtQFDCh3LXexnkANBDutgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7NSDsMfvZ/FvoJubuAR+fYy7XqnDmkqd18Ppg5Swwzj5D1UV5hK4oc8o56o1xjzyMbLScun+lvNkzFjPm4ImALCvcoQ/0Tupb+mFWIx0Qg7OuzxKsvvSFD4tpZ09xmgRxKZmAUDLCFVmjz2D1JCL2a1csZf9cNhOjzE7z6ct8MopPyH3YhXY4OfbZJPJ01/mK5ebgqpSdPM0X3wFRhTQT3GUP8Ly49Jza9AySkvuqVeM3Er+ct2FFBhN+L9okwGYyn9wts7l5blnmEx7TAPeve/G2nRS22y33Zct5/6K58bq6iRfoET2hp4jrFW3QW9pEFwXg/Ubf6cf8b2vJemA9LE0bcxXmPp17rMf/AG7jMm8=x2

°2 0 2°2

°1

0

1

2

circ

le

Input

°0.5 0.0 0.5°0.5

0.0

0.5

Shapley Representation

°2 0 2°2

°1

0

1

2Li

near£

sin

Input

°1 0 1

°1

0

1


°0.90°0.450.000.450.901.351.80

°0.90°0.450.000.450.901.351.80

°1.8°1.2°0.60.00.61.21.8

°3.2°2.4°1.6°0.80.00.81.62.43.2

°2 0 2°2

°1

0

1

2

circ

le

Input

0.5 1.0 1.5

0.5

1.0

1.5


°2 0 2°2

°1

0

1

2

Line

ar£

sin

Input

°1 0 1

°1

0

1


°0.90°0.450.000.450.901.351.80

°0.8°0.40.00.40.81.21.62.02.42.8

°1.8°1.2°0.60.00.61.21.8

°3.2°2.4°1.6°0.80.00.81.62.43.2

AAAGZXicpVTNTtRQFD6MUhFFQY0bFzZOSFzI2BKILIluXLhAZYAECWk7d6CZ2x/bW4WQeQS3+ga+k0+gvoXfOe0wAzNTSOzNdE7PPd937vm7fqrD3DjOr5nGjZuz1q252/N37i7cu7+49GAnT4osUO0g0Um253u50mGs2iY0Wu2lmfIiX6tdv/eG93e/qCwPk3jbnKbqIPKO4rAbBp6B6uPJoXu42HRajjz2uOBWQpOqZytZavykT9ShhAIqKCJFMRnImjzKsfbJJYdS6A7oDLoMUij7ivo0D2wBKwULD9oe3kf42q+0Mb6ZMxd0AC8avwxIm5aBSWCXQWZvtuwXwszaadxnwslnO8W/X3FF0Bo6hvYq3MDy+jgfVvWxGurShsQYIuZUNBx9UHkpJGscmT0StQFDCh3LHexnkANBDupgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7FSDsEfvp/NvoJubuAh9fYC7XsnDmkqcN8Ppg5Swwzj5H1UV5jK4oc8o56o5wDz0MbbScunetvNlTFjPm4ImALCvcpg/0Tupb+mFWIx0Qg7OuzxKsnvSFD4tJZ09xmgRxKZmAUDLCFVmhz2D1JCL2a1csZf9cNRPDzE7y6ct8MopPyH3YgXYwOfb5JPJ01/mK5ebgqpSdPMkX3wFRhTRj3GUP8Ly49JJW6QUkJfdVq8ZvJH45b4OK9Mf8XrZJgM1kPrlbpnPz3LLMJzyhPu5f9/JtOy7srLbc9Zbzfq25+bq6iefoCT2j54jrFW3SW9pCFwXg/Ubf6cfsb2vBemQ9Lk0bMxXmIV14rKf/AGjkMm4=x1AAAGdHicpVTLbtNQEJ0GGkp5pbCEhUUUqUhtalcguqxgw4JFQU1bqaki27lprPiFfV1aRfkTtvSf+AP4AdacGTtN2iRuJXwVZzx3zpk7r+vEvpdq0/y1VLl3f7n6YOXh6qPHT54+q609P0ijLHFVy438KDly7FT5Xqha2tO+OooTZQeOrw6dwUfePzxTSepF4b6+iNVJYJ+GXs9zbQ1Vp1Zrx32vY633NtpOMDwfvenU6mbTlMeYFaxCqFPx7EVrlUtqU5cicimjgBSFpCH7ZFOKdUwWmRRDd0JD6BJInuwrGtEqsBmsFCxsaAd4n+LruNCG+GbOVNAuvPj4JUAa1AAmgl0Cmb0Zsp8JM2sXcQ+Fk892gX+n4Aqg1dSH9jbc2PLuOAdW5bFq6tGOxOgh5lg0HL1beMkkaxyZMRW1BkMMHctd7CeQXUGO62AIJpXccO5t2f8tlqzlb7ewzehPaRTs05c4/icS9ulLFb9Dzu03gTCm7Bfz76ObmLsHfHiNOV8N4UwlTzvgdcDKWWCccYUqi7KPrshzyjnqTXFPPExsfDn14E55MxYsZkzBEwCZV7hFX+mz1Df3w6xaOiAEZ1mfRVgD6QsHFvPOHuM0EeJSMgGeZIQrsknfwGpLROzXKFjy/rltJiaZnefTkflkFJ+Q+7AL7XhyjKtJ5Oku8xXKzcFVyTt5ni++A4ICqWe48x7gebFoi7ZpA5KS+6pZ4jcQv5y3cUVGM35v2kTAJjKf3C2LuXluWeYTntMI969187adFQ62m9a7pvnlbX33Q3ETr9BLek3riOs97dIn2kMXuXRGP+gnXS7/rb6q1quN3LSyVGBe0LWn2vwHfkY3xA==

�1(f, x)

AAAGZXicpVTNbtNAEJ4Gakqh0ALiwgGLqBIHGuwIRI8VXDhwKNC0lUpV2c6mtbL+wV5DqyqPwBXegHfiCYC34Jux06RN4lbCqzjj2fm+2flbP9Vhbhzn11zj2vV568bCzcVbt5fu3F1eubedJ0UWqE6Q6CTb9b1c6TBWHRMarXbTTHmRr9WO33/D+ztfVJaHSbxlTlK1H3mHcdgLA89A9fH4oH2w3HRajjz2pOBWQpOqZzNZafykT9SlhAIqKCJFMRnImjzKsfbIJYdS6PbpFLoMUij7iga0CGwBKwULD9o+3of42qu0Mb6ZMxd0AC8avwxIm1aBSWCXQWZvtuwXwszaWdynwslnO8G/X3FF0Bo6gvYy3NDy6jgfVvWxGurRusQYIuZUNBx9UHkpJGscmT0WtQFDCh3LXexnkANBDutgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7NSDsMfvZ/FvoJubuAR+fYy7XqnDmkqd18Ppg5Swwzj5D1UV5hK4oc8o56o1xjzyMbLScun+lvNkzFjPm4ImALCvcoQ/0Tupb+mFWIx0Qg7OuzxKsvvSFD4tpZ09xmgRxKZmAUDLCFVmjz2D1JCL2a1csZf9cNhOjzE7z6ct8MopPyH3YhXY4OfbZJPJ01/mK5ebgqpSdPM0X3wFRhTQT3GUP8Ly49Jza9AySkvuqVeM3Er+ct2FFBhN+L9okwGYyn9wts7l5blnmEx7TAPeve/G2nRS22y33Zct5/6K58bq6iRfoET2hp4jrFW3QW9pEFwXg/Ubf6cf8b2vJemA9LE0bcxXmPp17rMf/AG7jMm8=x2

AAAGZXicpVTNTtRQFD6MUhFFQY0bFzZOSFzI2BKILIluXLhAZYAECWk7d6CZ2x/bW4WQeQS3+ga+k0+gvoXfOe0wAzNTSOzNdE7PPd937vm7fqrD3DjOr5nGjZuz1q252/N37i7cu7+49GAnT4osUO0g0Um253u50mGs2iY0Wu2lmfIiX6tdv/eG93e/qCwPk3jbnKbqIPKO4rAbBp6B6uPJoXu42HRajjz2uOBWQpOqZytZavykT9ShhAIqKCJFMRnImjzKsfbJJYdS6A7oDLoMUij7ivo0D2wBKwULD9oe3kf42q+0Mb6ZMxd0AC8avwxIm5aBSWCXQWZvtuwXwszaadxnwslnO8W/X3FF0Bo6hvYq3MDy+jgfVvWxGurShsQYIuZUNBx9UHkpJGscmT0StQFDCh3LHexnkANBDupgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7FSDsEfvp/NvoJubuAh9fYC7XsnDmkqcN8Ppg5Swwzj5H1UV5jK4oc8o56o5wDz0MbbScunetvNlTFjPm4ImALCvcpg/0Tupb+mFWIx0Qg7OuzxKsnvSFD4tJZ09xmgRxKZmAUDLCFVmhz2D1JCL2a1csZf9cNRPDzE7y6ct8MopPyH3YgXYwOfb5JPJ01/mK5ebgqpSdPMkX3wFRhTRj3GUP8Ly49JJW6QUkJfdVq8ZvJH45b4OK9Mf8XrZJgM1kPrlbpnPz3LLMJzyhPu5f9/JtOy7srLbc9Zbzfq25+bq6iefoCT2j54jrFW3SW9pCFwXg/Ubf6cfsb2vBemQ9Lk0bMxXmIV14rKf/AGjkMm4=x1

AAAGdHicpVTLTttQEB3SklL6Cu2yXViNIlEJghMVwRK1my66oBUBJIIi27khVvyqfU1BUf6k2/JP/YP2B7rumbFDAkkMUn0VZzx3zpk7r2tHnpto0/y1VHrwcLn8aOXx6pOnz56/qKy9PEzCNHZUywm9MD62rUR5bqBa2tWeOo5iZfm2p47swUfePzpXceKGwYG+jNSpb50Fbs91LA1Vp1JpR32301zvbbRtf3gxetepVM26KY8xKzRyoUr5sx+ula6oTV0KyaGUfFIUkIbskUUJ1gk1yKQIulMaQhdDcmVf0YhWgU1hpWBhQTvA+wxfJ7k2wDdzJoJ24MXDLwbSoBowIexiyOzNkP1UmFm7iHsonHy2S/zbOZcPraY+tHfhxpb3x9mwKo5VU492JUYXMUei4eid3EsqWePIjKmoNRgi6FjuYj+G7AhyXAdDMInkhnNvyf5vsWQtfzu5bUp/CqNgn57E8T+RsE9Pqvgdcma/CYQxZb+Y/wDdxNw94IMbzNmqCWciedoFrw1WzgLjjGtUUZR9dEWWU85Rb4p74mFi48mpB/fKm7FgMWMCHh/IrMIt+kqfpb6ZH2bV0gEBOIv6LMQaSF/YsJh39ginCRGXkglwJSNckU36BlZLImK/Rs6S9c9dMzHJ7Dyftswno/iE3IddaMeTY1xPIk93ka9Abg6uStbJ83zxHeDnSD3DnfUAz0uDtqhJG5CU3Ff1Ar+++OW8jSsymvF72yYENpb55G5ZzM1zyzKf8IJGuH8bt2/bWeGwWW9s180v76t7H/KbeIVe01taR1w7tEefaB9d5NA5/aCfdLX8t/ymXC3XMtPSUo55RTeecv0fhE83xQ==

�2(f, x)

AAAGdHicpVTLbtNQEJ0GGkp5pbCEhUUUqUhtalcguqxgw4JFQU1bqaki27lprPiFfV1aRfkTtvSf+AP4AdacGTtN2iRuJXwVZzx3zpk7r+vEvpdq0/y1VLl3f7n6YOXh6qPHT54+q609P0ijLHFVy438KDly7FT5Xqha2tO+OooTZQeOrw6dwUfePzxTSepF4b6+iNVJYJ+GXs9zbQ1Vp1Zrx32vY633NtpOMDwfvenU6mbTlMeYFaxCqFPx7EVrlUtqU5cicimjgBSFpCH7ZFOKdUwWmRRDd0JD6BJInuwrGtEqsBmsFCxsaAd4n+LruNCG+GbOVNAuvPj4JUAa1AAmgl0Cmb0Zsp8JM2sXcQ+Fk892gX+n4Aqg1dSH9jbc2PLuOAdW5bFq6tGOxOgh5lg0HL1beMkkaxyZMRW1BkMMHctd7CeQXUGO62AIJpXccO5t2f8tlqzlb7ewzehPaRTs05c4/icS9ulLFb9Dzu03gTCm7Bfz76ObmLsHfHiNOV8N4UwlTzvgdcDKWWCccYUqi7KPrshzyjnqTXFPPExsfDn14E55MxYsZkzBEwCZV7hFX+mz1Df3w6xaOiAEZ1mfRVgD6QsHFvPOHuM0EeJSMgGeZIQrsknfwGpLROzXKFjy/rltJiaZnefTkflkFJ+Q+7AL7XhyjKtJ5Oku8xXKzcFVyTt5ni++A4ICqWe48x7gebFoi7ZpA5KS+6pZ4jcQv5y3cUVGM35v2kTAJjKf3C2LuXluWeYTntMI969187adFQ62m9a7pvnlbX33Q3ETr9BLek3riOs97dIn2kMXuXRGP+gnXS7/rb6q1quN3LSyVGBe0LWn2vwHfkY3xA==

�1(f, x)

AAAGdHicpVTLTttQEB3SklL6Cu2yXViNIlEJghMVwRK1my66oBUBJIIi27khVvyqfU1BUf6k2/JP/YP2B7rumbFDAkkMUn0VZzx3zpk7r2tHnpto0/y1VHrwcLn8aOXx6pOnz56/qKy9PEzCNHZUywm9MD62rUR5bqBa2tWeOo5iZfm2p47swUfePzpXceKGwYG+jNSpb50Fbs91LA1Vp1JpR32301zvbbRtf3gxetepVM26KY8xKzRyoUr5sx+ula6oTV0KyaGUfFIUkIbskUUJ1gk1yKQIulMaQhdDcmVf0YhWgU1hpWBhQTvA+wxfJ7k2wDdzJoJ24MXDLwbSoBowIexiyOzNkP1UmFm7iHsonHy2S/zbOZcPraY+tHfhxpb3x9mwKo5VU492JUYXMUei4eid3EsqWePIjKmoNRgi6FjuYj+G7AhyXAdDMInkhnNvyf5vsWQtfzu5bUp/CqNgn57E8T+RsE9Pqvgdcma/CYQxZb+Y/wDdxNw94IMbzNmqCWciedoFrw1WzgLjjGtUUZR9dEWWU85Rb4p74mFi48mpB/fKm7FgMWMCHh/IrMIt+kqfpb6ZH2bV0gEBOIv6LMQaSF/YsJh39ginCRGXkglwJSNckU36BlZLImK/Rs6S9c9dMzHJ7Dyftswno/iE3IddaMeTY1xPIk93ka9Abg6uStbJ83zxHeDnSD3DnfUAz0uDtqhJG5CU3Ff1Ar+++OW8jSsymvF72yYENpb55G5ZzM1zyzKf8IJGuH8bt2/bWeGwWW9s180v76t7H/KbeIVe01taR1w7tEefaB9d5NA5/aCfdLX8t/ymXC3XMtPSUo55RTeecv0fhE83xQ==

�2(f, x)AAAGZXicpVTNbtNAEJ4Gakqh0ALiwgGLqBIHGuwIRI8VXDhwKNC0lUpV2c6mtbL+wV5DqyqPwBXegHfiCYC34Jux06RN4lbCqzjj2fm+2flbP9Vhbhzn11zj2vV568bCzcVbt5fu3F1eubedJ0UWqE6Q6CTb9b1c6TBWHRMarXbTTHmRr9WO33/D+ztfVJaHSbxlTlK1H3mHcdgLA89A9fH4oH2w3HRajjz2pOBWQpOqZzNZafykT9SlhAIqKCJFMRnImjzKsfbIJYdS6PbpFLoMUij7iga0CGwBKwULD9o+3of42qu0Mb6ZMxd0AC8avwxIm1aBSWCXQWZvtuwXwszaWdynwslnO8G/X3FF0Bo6gvYy3NDy6jgfVvWxGurRusQYIuZUNBx9UHkpJGscmT0WtQFDCh3LXexnkANBDutgCyaX3HDuPdn/I5as5e+gsi3ob20U7FNLHP8TCfvUUsWvkEv7NSDsMfvZ/FvoJubuAR+fYy7XqnDmkqd18Ppg5Swwzj5D1UV5hK4oc8o56o1xjzyMbLScun+lvNkzFjPm4ImALCvcoQ/0Tupb+mFWIx0Qg7OuzxKsvvSFD4tpZ09xmgRxKZmAUDLCFVmjz2D1JCL2a1csZf9cNhOjzE7z6ct8MopPyH3YhXY4OfbZJPJ01/mK5ebgqpSdPM0X3wFRhTQT3GUP8Ly49Jza9AySkvuqVeM3Er+ct2FFBhN+L9okwGYyn9wts7l5blnmEx7TAPeve/G2nRS22y33Zct5/6K58bq6iRfoET2hp4jrFW3QW9pEFwXg/Ubf6cf8b2vJemA9LE0bcxXmPp17rMf/AG7jMm8=x2

AAAGhnicpVRLTxNRFD6g1ooPii7d3NCQYAJ1hkggrhrduHCBhAIJEDIzvYVJ5+XMHYQ03ftLXLrVv+I/0H/hd85MabEPSJybTs+ce77v3PO6bhL4mbGsX3Pz9+4/qDysPlp4/OTps8Xa0vP9LM5TT7e8OIjTQ9fJdOBHumV8E+jDJNVO6Ab6wO2+5/2DC51mfhztmatEn4TOWeR3fM8xUJ3Wlo+NvjQ9ebud3q7uNNSFE+Q6679V/VV7zX51WqvbDUseZY0Jg606lc9OvDT/jY6pTTF5lFNImiIykANyKMM6IpssSqA7oR50KSRf9jX1aQHYHFYaFg60XbzP8HVUaiN8M2cmaA9eAvxSIBWtABPDLoXM3pTs58LM2mncPeHks13h3y25QmgNnUN7G25geXecC6vZsRrq0LbE6CPmRDQcvVd6ySVrHJkaidqAIYGO5Tb2U8ieIAd1UILJJDece0f2f4sla/nbK21z+jMzCvYZSBz/Ewn7DKSKXyAX9utAqBH76fx76Cbm7gAf3WAu1opwZpKnbfC6YOUsME5do2ZFeY6uKHLKOeqMcA89DG0COXX3TnlTUxYzZuAJgSwq3KJd+ij1Lfwwq5EOiMA5q89irK70hQuLSWdPcJoYcWmZAF8ywhVZp89gdSQi9qtKlqJ/bpuJYWYn+XRlPhnFJ+Q+bEM7mBx1PYk83bN8RXJzcFWKTp7ki++AsESaMe6iB3hebHpNG7QGSct91ZjhNxS/nLdBRfpjfv+1iYFNZT65W6Zz89yyzCe8pP7o/Ttd2N9o2JsN69ObevNdeRNX6SUt0yri2qImfaAddJFHX+k7/aCflWqlUdmsbBWm83Ml5gXdeCrNvw6uPf0=

Ref. values: (1, 1)AAAGhnicpVTLTttQEB1om6b0QWiX3VhESFQC10FFoK6idtNFFxQRQAKEbOcarPhV+5qCouz7JV122/5K/6D9i54ZOyQ0iUGqr+KM5845c+d1nSTwM21Zv+bm791/UHtYf7Tw+MnTZ4uNpef7WZynruq4cRCnh46dqcCPVEf7OlCHSars0AnUgdN7z/sHFyrN/Dja01eJOgnts8j3fNfWUJ02lo+1utR9eTtef1d5pnFhB7nKBm+Nwaq11np12mi2TEsew5oQhltNKp+deGn+Gx1Tl2JyKaeQFEWkIQdkU4Z1RC2yKIHuhPrQpZB82Vc0oAVgc1gpWNjQ9vA+w9dRqY3wzZyZoF14CfBLgTRoBZgYdilk9mbIfi7MrJ3F3RdOPtsV/p2SK4RW0zm0t+GGlnfHObCqjlWTR9sSo4+YE9Fw9G7pJZescWTGWNQaDAl0LHexn0J2BTmsgyGYTHLDubdl/7dYspa/3dI2pz+VUbDPQOL4n0jYZyBV/AK5sF8Hwhizn82/h25ibg/46AZzsVaEM5M8bYPXAStngXHGNaoqynN0RZFTzpE3xj3yMLIJ5NS9O+XNmLGYMQNPCGRR4Q7t0kepb+GHWbV0QATOqj6LsXrSFw4spp09wWlixKVkAnzJCFdknT6D1ZaI2K9RshT9c9tMjDI7zacj88koPiH3YRfa4eQY15PI013lK5Kbg6tSdPI0X3wHhCVST3AXPcDz0qLXtEFrkJTcV2aF31D8ct6GFRlM+P3XJgY2lfnkbpnNzXPLMp/wkgbj9+9sYX/DbG2a1qc3zfa78iau00taplXEtUVt+kA76CKXvtJ3+kE/a/WaWdusbRWm83Ml5gXdeGrtvwisPfw=

Ref. values: (0, 1)

Figure 4: Shapley representation of two-dimensional functions. Such representation can span beyondone dimensional manifold and depends on both the inner function and the reference values. In bothgroups, the gray-scale background indicates the respective function value while the rainbow-scalecolor indicates correspondence between input (left) and Shapley representation (right) along withfunction values—red means highest and purple means lowest function values. The red cross in theinput plots represents the reference values for both inputs. More details in subsection 2.1.

A Three SHAP properties

The first property called local accuracy states that the approximate model D at z = 1 should matchthe output of the model C at the corresponding x,

Definition 12 (Local Accuracy). Suppose C is the true model, x is any target point, D is the modelexplanation as a simplified model and z is the simplified binary vector corresponding to x, localaccuracy is the property such that

C(x) = D(z) = a0 +

d∑

i=1

aizi .

The second property called missingness formalizes the idea that features that are “missing” from theinput x (or correspondingly the zeros in z) should have zero attributed effect on the output of theapproximation,i

Definition 13 (Missingness).

zi = 0⇒ ai = 0 .

Finally, the third property called consistency formalizes the idea that if one model always sees alarger effect when removing a feature, the attribution value should be larger [1].

B Notes on SHAPNET

Remark 14 (Row-wise sparsity curbs computation). One of the main bottleneck of computingShapley values is the exponential complexity in the number of input features that renders large-scalecomputing of Shapley values infeasible. To alleviate such issue, we would like to promote thesparsity in row-wise computation. Consider a function f depending on a subset of input features,(e.g. f (j)(x1, x2, x3) = 5x2 + x3 only depends on x2 and x3), we will refer to such a subsetactive set , A(f) (A(f (j)) = {2, 3} for the example above) and its complement dummy set, D(f)(D(f (j)) = {1} for the above example). In principle, we would like to keep the cardinality of theactive set low:

∣∣A(f (j))∣∣ � d for 1 ≤ j ≤ c, s.t. we can limit the computational overhead, as

features in the dummy set will be trivially assigned Shapley values of 0’s.

Remark 15 (Vector-valued functions as tuples). Def. 2 & Def. 3 can be trivially extended tovector-valued functions: f : Rd → Rc, where we simply view each output as a scalar function, i.e.,f (j)(x) ≡ [f(x)]j . As an example, suppose that f is a multi-class classifier with c classes, then eachof the f (j) would be the logits for the j-th class.

Remark 16 (Tuples of vector-valued functions: the meta-channels). Suppose we are to compute theShapley Transform of a tuple of vector-valued functions (f (1)(x), · · · , f (c)(x)) where f (j) : Rd 7→Rn, we will need to compute φi(f (j),x) for each output of f (j). In this case, we have again createdanother dimension as φi(f (j),x) now would become a vector instead of remaining as a scalar (by

9

AAACx3icjVHLTsJAFD3UF+ILdemmkZi4Iq0h0SXRDUuMIiRISDsM2FDaZjolEOLCD3Crn2b8A/0L74xDohKj07Q9c+49Z+be6ydhkErHec1ZS8srq2v59cLG5tb2TnF37yaNM8F4g8VhLFq+l/IwiHhDBjLkrURwb+SHvOkPL1S8OeYiDeLoWk4T3hl5gyjoB8yTRF1Num63WHLKjl72InANKMGselx8wS16iMGQYQSOCJJwCA8pPW24cJAQ18GMOEEo0HGOexRIm1EWpwyP2CF9B7RrGzaivfJMtZrRKSG9gpQ2jkgTU54grE6zdTzTzor9zXumPdXdpvT3jdeIWIk7Yv/SzTP/q1O1SPRxpmsIqKZEM6o6Zlwy3RV1c/tLVZIcEuIU7lFcEGZaOe+zrTWprl311tPxN52pWLVnJjfDu7olDdj9Oc5F0Dwpu5Wy615WStVzM+s8DnCIYxroKaqooY4GmQ/wiCc8WzUrssbW5DPVyhnNPr4t6+EDp96QWg==x1

AAACx3icjVHLTsJAFD3UF+ILdemmkZi4Ii0x0SXRDUuMIiRISDsMOKG0zXRKIMSFH+BWP834B/oX3hlLohKj07Q9c+49Z+be68eBSJTjvOaspeWV1bX8emFjc2t7p7i7d5NEqWS8waIgki3fS3ggQt5QQgW8FUvujfyAN/3hhY43x1wmIgqv1TTmnZE3CEVfME8RdTXpVrrFklN2zLIXgZuBErJVj4ovuEUPERhSjMARQhEO4CGhpw0XDmLiOpgRJwkJE+e4R4G0KWVxyvCIHdJ3QLt2xoa0156JUTM6JaBXktLGEWkiypOE9Wm2iafGWbO/ec+Mp77blP5+5jUiVuGO2L9088z/6nQtCn2cmRoE1RQbRlfHMpfUdEXf3P5SlSKHmDiNexSXhJlRzvtsG01iate99Uz8zWRqVu9ZlpviXd+SBuz+HOciaFbK7knZdS9PStXzbNZ5HOAQxzTQU1RRQx0NMh/gEU94tmpWaI2tyWeqlcs0+/i2rIcPqj+QWw==x2

AAACx3icjVHLTsJAFD3UF+ILdemmkZi4Ii0h0SXRDUuMIiRISDsM2FDaZjolEuLCD3Crn2b8A/0L74xDohKj07Q9c+49Z+be6ydhkErHec1ZS8srq2v59cLG5tb2TnF37zqNM8F4k8VhLNq+l/IwiHhTBjLk7URwb+yHvOWPzlW8NeEiDeLoSk4T3h17wygYBMyTRF2KXqVXLDllRy97EbgGlGBWIy6+4AZ9xGDIMAZHBEk4hIeUng5cOEiI62JGnCAU6DjHPQqkzSiLU4ZH7Ii+Q9p1DBvRXnmmWs3olJBeQUobR6SJKU8QVqfZOp5pZ8X+5j3TnupuU/r7xmtMrMQtsX/p5pn/1alaJAY41TUEVFOiGVUdMy6Z7oq6uf2lKkkOCXEK9ykuCDOtnPfZ1ppU16566+n4m85UrNozk5vhXd2SBuz+HOciaFXKbrXsuhfVUu3MzDqPAxzimAZ6ghrqaKBJ5kM84gnPVt2KrIl195lq5YxmH9+W9fABm+2QVQ==r2

AAACx3icjVHLSsNAFD2Nr1pfVZdugkVwVRIp6LLopsuK1gq1lGQ6raF5MZkUS3HhB7jVTxP/QP/CO+MU1CI6IcmZc+85M/dePw2DTDrOa8FaWFxaXimultbWNza3yts7V1mSC8ZbLAkTce17GQ+DmLdkIEN+nQruRX7I2/7oTMXbYy6yIIkv5STl3cgbxsEgYJ4k6kL03F654lQdvex54BpQgVnNpPyCG/SRgCFHBI4YknAIDxk9HbhwkBLXxZQ4QSjQcY57lEibUxanDI/YEX2HtOsYNqa98sy0mtEpIb2ClDYOSJNQniCsTrN1PNfOiv3Ne6o91d0m9PeNV0SsxC2xf+lmmf/VqVokBjjRNQRUU6oZVR0zLrnuirq5/aUqSQ4pcQr3KS4IM62c9dnWmkzXrnrr6fibzlSs2jOTm+Nd3ZIG7P4c5zxoH1XdWtV1z2uV+qmZdRF72MchDfQYdTTQRIvMh3jEE56thhVbY+vuM9UqGM0uvi3r4QOZjJBUr1

AAACx3icjVHLTsJAFD3UF+ILdemmkZi4Iq0h0SXRDUuMIiRISDsM2FDaZjolEOLCD3Crn2b8A/0L74xDohKj07Q9c+49Z+be6ydhkErHec1ZS8srq2v59cLG5tb2TnF37yaNM8F4g8VhLFq+l/IwiHhDBjLkrURwb+SHvOkPL1S8OeYiDeLoWk4T3hl5gyjoB8yTRF1Num63WHLKjl72InANKMGselx8wS16iMGQYQSOCJJwCA8pPW24cJAQ18GMOEEo0HGOexRIm1EWpwyP2CF9B7RrGzaivfJMtZrRKSG9gpQ2jkgTU54grE6zdTzTzor9zXumPdXdpvT3jdeIWIk7Yv/SzTP/q1O1SPRxpmsIqKZEM6o6Zlwy3RV1c/tLVZIcEuIU7lFcEGZaOe+zrTWprl311tPxN52pWLVnJjfDu7olDdj9Oc5F0Dwpu5Wy615WStVzM+s8DnCIYxroKaqooY4GmQ/wiCc8WzUrssbW5DPVyhnNPr4t6+EDp96QWg==x1AAACx3icjVHLTsJAFD3UF+ILdemmkZi4Ii0x0SXRDUuMIiRISDsMOKG0zXRKIMSFH+BWP834B/oX3hlLohKj07Q9c+49Z+be68eBSJTjvOaspeWV1bX8emFjc2t7p7i7d5NEqWS8waIgki3fS3ggQt5QQgW8FUvujfyAN/3hhY43x1wmIgqv1TTmnZE3CEVfME8RdTXpVrrFklN2zLIXgZuBErJVj4ovuEUPERhSjMARQhEO4CGhpw0XDmLiOpgRJwkJE+e4R4G0KWVxyvCIHdJ3QLt2xoa0156JUTM6JaBXktLGEWkiypOE9Wm2iafGWbO/ec+Mp77blP5+5jUiVuGO2L9088z/6nQtCn2cmRoE1RQbRlfHMpfUdEXf3P5SlSKHmDiNexSXhJlRzvtsG01iate99Uz8zWRqVu9ZlpviXd+SBuz+HOciaFbK7knZdS9PStXzbNZ5HOAQxzTQU1RRQx0NMh/gEU94tmpWaI2tyWeqlcs0+/i2rIcPqj+QWw==x2

AAAGZnicpVTNThNRFD5UqYiioFEXbm4kJC6kzgCJLAluXLgApZYECZmZ3sKk8+fMHYWQPoJbfQN3voUv4cat+hZ+58yUFtoOJM5Np2fOPd937vm7bhL4mbGsn1O1a9en6zdmbs7euj135+78wr13WZynnm56cRCnu66T6cCPdNP4JtC7Saqd0A10y+2+5P3WR51mfhztmJNE74fOYeR3fM8xUL09Plg9mF+0GpY8alSwS2Fx4+H2L//75o+teKH2jd5Tm2LyKKeQNEVkIAfkUIa1RzZZlEC3T6fQpZB82dfUo1lgc1hpWDjQdvE+xNdeqY3wzZyZoD14CfBLgVS0BEwMuxQye1Oynwszaydxnwonn+0E/27JFUJr6Ajay3B9y6vjXFhVx2qoQ+sSo4+YE9Fw9F7pJZescWRqKGoDhgQ6ltvYTyF7guzXQQkmk9xw7h3Z/yOWrOVvr7TN6W9lFOwzkDj+JxL2GUgVP0Eu7JeBUEP2k/l30E3M3QE+OsdcrCXhzCRP6+B1wcpZYJw6Q1VFeYSuKHLKOeoMcQ88DGwCOXX3SnlTExYzZuAJgSwq3KQ39FrqW/hhViMdEIGzqs9irK70hQuLcWdPcJoYcWmZAF8ywhVZpg9gdSQi9qtKlqJ/LpuJQWbH+XRlPhnFJ+Q+bEPbnxx1Nok83VW+Irk5uCpFJ4/zxXdAWCLNCHfRAzwvNj2nFXoGSct91ajwG4pfzlu/Ir0RvxdtYmBTmU/ulsncPLcs8wmPqYf71754244KrZWGvdaw7W3cxJtUPDP0mJ7QUwT2gjboFW2hjTwQf6Yv9HX6d32u/qD+qDCtTZWY+3Tuqat/1hQ2Qg==x3AAAGZnicpVTNThNRFD5UqYiioFEXbm4kJC6kzhASWBLcuHABSi0JEjIzvYVJ58+ZOwghfQS3+gbufAtfwo1b9S38zpkpLbQdSJybTs+ce77v3PN33STwM2NZP6dqN25O12/N3J69c3fu3v35hQfvszhPPd304iBOd10n04Ef6abxTaB3k1Q7oRvoltt9xfutY51mfhztmNNE74fOYeR3fM8xUL07OVg9mF+0GpY8alSwS2Fx4/H2L//75o+teKH2jT5Qm2LyKKeQNEVkIAfkUIa1RzZZlEC3T2fQpZB82dfUo1lgc1hpWDjQdvE+xNdeqY3wzZyZoD14CfBLgVS0BEwMuxQye1Oynwszaydxnwknn+0U/27JFUJr6Ajaq3B9y+vjXFhVx2qoQ+sSo4+YE9Fw9F7pJZescWRqKGoDhgQ6ltvYTyF7guzXQQkmk9xw7h3Z/yOWrOVvr7TN6W9lFOwzkDj+JxL2GUgVP0Eu7JeBUEP2k/l30E3M3QE+usBcrCXhzCRP6+B1wcpZYJw6R1VFeYSuKHLKOeoMcQ88DGwCOXX3WnlTExYzZuAJgSwq3KS39EbqW/hhViMdEIGzqs9irK70hQuLcWdPcJoYcWmZAF8ywhVZpo9gdSQi9qtKlqJ/rpqJQWbH+XRlPhnFJ+Q+bEPbnxx1Pok83VW+Irk5uCpFJ4/zxXdAWCLNCHfRAzwvNr2kFXoBSct91ajwG4pfzlu/Ir0Rv5dtYmBTmU/ulsncPLcs8wlPqIf71758244KrZWGvdqw7W3cxJtUPDP0lJ7RcwS2Rhv0mrbQRh6IP9MX+jr9uz5Xf1R/UpjWpkrMQ7rw1NU/3BQ2Qw==x4

AAACx3icjVHLTsJAFD3UF+ILdemmkZi4Iq0h0SXRDUuMIiRISDsM2FDaZjolEOLCD3Crn2b8A/0L74xDohKj07Q9c+49Z+be6ydhkErHec1ZS8srq2v59cLG5tb2TnF37yaNM8F4g8VhLFq+l/IwiHhDBjLkrURwb+SHvOkPL1S8OeYiDeLoWk4T3hl5gyjoB8yTRF1Num63WHLKjl72InANKMGselx8wS16iMGQYQSOCJJwCA8pPW24cJAQ18GMOEEo0HGOexRIm1EWpwyP2CF9B7RrGzaivfJMtZrRKSG9gpQ2jkgTU54grE6zdTzTzor9zXumPdXdpvT3jdeIWIk7Yv/SzTP/q1O1SPRxpmsIqKZEM6o6Zlwy3RV1c/tLVZIcEuIU7lFcEGZaOe+zrTWprl311tPxN52pWLVnJjfDu7olDdj9Oc5F0Dwpu5Wy615WStVzM+s8DnCIYxroKaqooY4GmQ/wiCc8WzUrssbW5DPVyhnNPr4t6+EDp96QWg==x1AAACx3icjVHLTsJAFD3UF+ILdemmkZi4Ii0x0SXRDUuMIiRISDsMOKG0zXRKIMSFH+BWP834B/oX3hlLohKj07Q9c+49Z+be68eBSJTjvOaspeWV1bX8emFjc2t7p7i7d5NEqWS8waIgki3fS3ggQt5QQgW8FUvujfyAN/3hhY43x1wmIgqv1TTmnZE3CEVfME8RdTXpVrrFklN2zLIXgZuBErJVj4ovuEUPERhSjMARQhEO4CGhpw0XDmLiOpgRJwkJE+e4R4G0KWVxyvCIHdJ3QLt2xoa0156JUTM6JaBXktLGEWkiypOE9Wm2iafGWbO/ec+Mp77blP5+5jUiVuGO2L9088z/6nQtCn2cmRoE1RQbRlfHMpfUdEXf3P5SlSKHmDiNexSXhJlRzvtsG01iate99Uz8zWRqVu9ZlpviXd+SBuz+HOciaFbK7knZdS9PStXzbNZ5HOAQxzTQU1RRQx0NMh/gEU94tmpWaI2tyWeqlcs0+/i2rIcPqj+QWw==x2


AAAGfHicpVRLaxNRFD6NNtb6SnXhQpGLoSBoYqaIdlkUwYWLVhpTaEuZmdwkQ+blzI1tCVn6SwRXuu9f8R8o/gbB75yZNGnzaMG5ZHLm3PN9557XdWLfS02t9nOhcOXqYvHa0vXlGzdv3b5TWrn7MY16iavrbuRHyY5jp9r3Ql03nvH1TpxoO3B83XC6b3i/8VknqReF2+Y41vuB3Q69lufaBqqD0v09o49M/22zrdWh9todo6KWGlgHpXKtWpNHTQpWLpQ3Hp5s/fny6GQzWil8oz1qUkQu9SggTSEZyD7ZlGLtkkU1iqHbpz50CSRP9jUNaBnYHqw0LGxou3i38bWba0N8M2cqaBdefPwSIBWtAhPBLoHM3pTs94SZtbO4+8LJZzvGv5NzBdAa6kB7EW5oeXmcA6v5sRpq0brE6CHmWDQcvZt76UnWODI1FrUBQwwdy03sJ5BdQQ7roASTSm4497bs/xJL1vK3m9v26PfcKNinL3H8TyTs05cqHkLO7CtAqDH72fzb6CbmbgEfnmHO1qpwppKndfA6YOUsME6douZF2UFXZDnlHLXGuEceRja+nLp7qbypGYsZU/AEQGYVrtMHei/1zfwwq5EOCME5r88irK70hQOLaWePcZoIcWmZAE8ywhWp0Cew2hIR+1U5S9Y/F83EKLPTfDoyn4ziE3IfNqEdTo46nUSe7nm+Qrk5uCpZJ0/zxXdAkCPNBHfWAzwvFj2nNXoGSct9VZ3jNxC/nLdhRQYTfs/bRMAmMp/cLbO5eW5Z5hMe0QD3r3X+tp0UGmtV60XVsrZwE7+m7FmiB/SYniCwV7RB72gTbcQp+krf6cfi32K5+LRYyUwLCznmHp15ii//AWKsPrc=

Edge weight of 1

AAAGeXicpVTNThNRFD5UqYg/FHWnMTcSEmKkdtBElkQ3LlyAoZYECJmZ3tJJ58+ZWwSbLn0PVyb6BLyKb6DxDVz5nTNTWmg7kDg3nZ4593zfuefvOrHvpaZW+zlTunZ9tnxj7ub8rdt37i5UFu99SKNu4uq6G/lRsuPYqfa9UNeNZ3y9EyfaDhxfN5zOG95vHOkk9aJw25zEej+wD0Ov5bm2geqg8mDP6GPTa2gVJ9GRVqZtm/5BZalWrcmjxgUrF5Y2Hp1u/fny+HQzWix9oz1qUkQudSkgTSEZyD7ZlGLtkkU1iqHbpx50CSRP9jX1aR7YLqw0LGxoO3gf4ms314b4Zs5U0C68+PglQCpaBiaCXQKZvSnZ7woza6dx94STz3aCfyfnCqA11Ib2MtzA8uo4B1bFsRpq0brE6CHmWDQcvZt76UrWODI1ErUBQwwdy03sJ5BdQQ7qoASTSm4497bs/xJL1vK3m9t26XdhFOzTlzj+JxL26UsVP0HO7FeBUCP20/m30U3M3QI+PMecrWXhTCVP6+B1wMpZYJw6QxVF2UZXZDnlHLVGuIcehja+nLpzpbypKYsZU/AEQGYVrtN7eif1zfwwq5EOCMFZ1GcRVkf6woHFpLPHOE2EuLRMgCcZ4Yqs0kew2hIR+1U5S9Y/l83EMLOTfDoyn4ziE3IfNqEdTI46m0Se7iJfodwcXJWskyf54jsgyJFmjDvrAZ4Xi57TGj2DpOW+qhb4DcQv521Qkf6Y34s2EbCJzCd3y3RunluW+YTHxPevdfG2HRcaa1XrZdWytnATv6bsmaOH9IRWENgr2qC3tIk2cukzfaXv9GP2b1mVV8pPM9PSTI65T+ee8ot/kjw+Dw==

We prove that

AAAGh3icpVRLb9NAEJ4GCKG8UjggxGXVqhIHGuLwCrcCQuLAoaCGVGqrynY2iRW/sDe0VZQ7v4QjV/gZXOHCtfwLvhk7TUqTtBJexRnPzvfNzmud2PdSU63+XChcuHipeLl0ZfHqtes3bpaXbn1Io37i6oYb+VGy5dip9r1QN4xnfL0VJ9oOHF83nd4r3m9+0knqReGmOYz1bmB3Qq/tubaBaq+8vGP0gRm8bnW02tdep2tU1FZDtdNObHdgDQe14V55pVqpyqMmhOf1uvXkkbJyzcp6/fePO7+O7m5ES4UvtEMtisilPgWkKSQD2SebUqxtsqhKMXS7NIAugeTJvqYhLQLbh5WGhQ1tD+8OvrZzbYhv5kwF7cKLj18CpKJVYCLYJZDZm5L9vjCzdhb3QDj5bIf4d3KuAFpDXWjPwo0sz49zYDU/VkNtqkuMHmKORcPRu7mXvmSNI1MTURswxNCx3MJ+AtkV5KgOSjCp5IZzb8v+kViylr/d3LZPf+ZGwT59ieN/ImGfvlRxH3JmvwaEmrCfzb+JbmLuNvDhCeZsrQpnKnmqg9cBK2eBceoYNS/KLroiyynnqD3BPfYwtvHl1L1z5U3NWMyYgicAMqtwg97TW6lv5odZjXRACM55fRZh9aQvHFhMO3uM00SIS8sEeJIRrsgafQSrLRGxX5WzZP1z1kyMMzvNpyPzySg+IfdhC9rR5KjjSeTpnucrlJuDq5J18jRffAcEOdKc4s56gOfFoodUoweQtNxXlTl+A/HLeRtVZHjK7782EbCJzCd3y2xunluW+YQHxPfv6JJVs4VmrWI9rljWO9zELyl7SnSPluk+AntG6/SGNtBGLn2mr/SNvhdLxUrxabGemRYWcsxtOvEUX/wFIUVC8A==

Edge weight of1

2AAAGiHicpVTNThNRFD6gVsA/0IUxbm4kJC6kzFQiZUcwJi5coKFCQomZmd62k86fM7cCafoAvohbt/oWbnXjFt/C75yZ0iJtIXFuOj1z7vm+c8/fdZPAz4xl/ZyZvXb9Runm3PzCrdt37t5bXLr/Pou7qadrXhzE6b7rZDrwI10zvgn0fpJqJ3QDved2XvL+3iedZn4c7ZqTRB+GTivym77nGKg+LC7XjT42vVeNllZH2m+1jYqbqq9W683U8Xp2v1fpw8oqb1Zf2PZzZZUteVioVivWurILzfJW9fePh79OH+3ES7NfqE4NismjLoWkKSIDOSCHMqwDssmiBLpD6kGXQvJlX1OfFoDtwkrDwoG2g3cLXweFNsI3c2aC9uAlwC8FUtEKMDHsUsjsTcl+V5hZO4m7J5x8thP8uwVXCK2hNrSX4QaWV8e5sJoeq6EmVSVGHzEnouHovcJLV7LGkamRqA0YEuhYbmA/hewJclAHJZhMcsO5d2T/VCxZy99eYdulP1OjYJ+BxPE/kbDPQKp4BDm3XwVCjdhP5t9FNzF3E/joHHO+VoQzkzxVweuClbPAOHWGmhZlG12R55Rz1BzhHnoY2gRy6s6V8qYmLGbMwBMCmVe4Ru/ojdQ398OsRjogAue0PouxOtIXLizGnT3BaWLEpWUCfMkIV2SVPoLVkYjYrypY8v65bCaGmR3n05X5ZBSfkPuwAe1gctTZJPJ0T/MVyc3BVck7eZwvvgPCAmkucOc9wPNi0xpV6BkkLfdVeYrfUPxy3gYV6V/w+69NDGwq88ndMpmb55ZlPuEx8f07uGTVZGGvUrbXy7b9FjfxNuXPHD2mJ/QUgW3QFr2mHbSRR5/pK32j76X50lppo7SZm87OFJgHdO4pbf8Fw8lDNw==

Edge weight of � 12


AAAGenicpVTLbtNQEJ0GGkp5pSCxYWOlqpSobWpXILqMYMOCRUFNW6kpke3eNCZ+4Qc0CvkLvoAFG/ih/gH8AwvOjJ0mbR6thK/ijOfOOXPnda3QdeJE188XCrduLxbvLN1dvnf/wcNHpZXH+3GQRrZq2IEbRIeWGSvX8VUjcRJXHYaRMj3LVQdW9zXvH3xWUewE/l7SC9WxZ576TtuxzQSqVulpM+w4LaPS/tCvfKwONpqW1z8bVFulVb2my6NNCkYurNbLzfVv5/XebrBS+EFNOqGAbErJI0U+JZBdMinGOiKDdAqhO6Y+dBEkR/YVDWgZ2BRWChYmtF28T/F1lGt9fDNnLGgbXlz8IiA1WgMmgF0Emb1psp8KM2tncfeFk8/Ww7+Vc3nQJtSB9jrc0PLmOAtW82NNqE07EqODmEPRcPR27iWVrHFk2ljUCRhC6Fg+wX4E2RbksA6aYGLJDefelP3fYsla/rZz25T+zI2CfboSx/9Ewj5dqeIXyJn9JhDamP1s/j10E3O3gfcvMWdrTThjydMOeC2wchYYp12g5kXZQVdkOeUctce4Rx5GNq6cunujvGkzFjPG4PGAzCrcoPf0Vuqb+WHWRDrAB+e8PguwutIXFiymnT3EaQLEpWQCHMkIV2STPoHVlIjYr5azZP1z3UyMMjvNpyXzySg+IffhCbTDydEuJpGne54vX24OrkrWydN88R3g5chkgjvrAZ4Xg7ZomzYgKbmvanP8euKX8zasyGDC71WbANhI5pO7ZTY3zy3LfMIzGuD+Na7etpPC/nbNeFHT3+EifkXZs0TPqEwVxPWS6vSGdtFFNn2l7/STfi3+LZaL1eJ6ZlpYyDFP6NJTfP4P0o89HA==

�1(f(j), x)

AAAGenicpVTLbtNQEJ0GGkp5pSCxYWOlqpSobepEILqMYMOCRUFNW6kpke3cNCZ+4Qc0CvkLvoAFG/ih/gH8AwvOjJ0mbR6thK/ijOfOOXPndc3AsaNY18+XcrduL+fvrNxdvXf/wcNHhbXHB5GfhJZqWL7jh0emESnH9lQjtmNHHQWhMlzTUYdm7zXvH35WYWT73n7cD9SJa5x6dse2jBiqVuFpM+jarVqp82FQ+lgebjVNd3A2LLcK63pFl0ebFqqZsF4vNje/ndf7e/5a7gc1qU0+WZSQS4o8iiE7ZFCEdUxV0imA7oQG0IWQbNlXNKRVYBNYKVgY0PbwPsXXcab18M2ckaAteHHwC4HUaAMYH3YhZPamyX4izKydxz0QTj5bH/9mxuVCG1MX2utwI8ub40xYLY41pg7tSow2Yg5Ew9FbmZdEssaRaRNRx2AIoGO5jf0QsiXIUR00wUSSG869Ifu/xZK1/G1ltgn9WRgF+3Qkjv+JhH06UsUvkFP7bSC0Cfv5/PvoJubuAO9dYk7XhnBGkqdd8Jpg5SwwTrtALYqyi65Ic8o56kxwjz2MbRw5de9GedPmLGaMwOMCmVa4Qe/prdQ39cOssXSAB85FfeZj9aQvTFjMOnuA0/iIS8kE2JIRrsg2fQKrIRGxXy1jSfvnupkYZ3aWT1Pmk1F8Qu7DNrSjydEuJpGne5EvT24OrkraybN88R3gZsh4ijvtAZ6XKu1QjbYgKbmvKgv8uuKX8zaqyHDK71UbH9hQ5pO7ZT43zy3LfMIzGuL+rV69baeFg1ql+qKiv8NF/IrSZ4WeUZFKiOsl1ekN7aGLLPpK3+kn/Vr+my/my/nN1DS3lGGe0KUn//wf2J49HQ==

�2(f(j), x)

AAAGenicpVTLbtNQEJ0GGkp5pSCxYWOlqpSobXAKiC4j2LBgUVDTVmpKZLs3jYlf+AGNQv6CL2DBBn6ofwD/wIIzY6dJm8SthK/ijOfOOXPndc3AsaNY188WCjduLhZvLd1evnP33v0HpZWHe5GfhJZqWr7jhwemESnH9lQztmNHHQShMlzTUftm7zXv739WYWT73m7cD9SRa5x4dse2jBiqdulxK+ja7WeVzodB5WN1uNEy3cHpsNoureo1XR5tWqhnwmqj3Fr/dtbo7/grhR/UomPyyaKEXFLkUQzZIYMirEOqk04BdEc0gC6EZMu+oiEtA5vASsHCgLaH9wm+DjOth2/mjARtwYuDXwikRmvA+LALIbM3TfYTYWbtPO6BcPLZ+vg3My4X2pi60F6FG1leH2fCKj/WmDq0LTHaiDkQDUdvZV4SyRpHpk1EHYMhgI7lY+yHkC1BjuqgCSaS3HDuDdn/LZas5W8rs03oT24U7NOROP4nEvbpSBW/QE7tN4HQJuzn8++im5i7A7x3gTlda8IZSZ62wWuClbPAOO0clRdlF12R5pRz1JngHnsY2zhy6t618qbNWcwYgccFMq1wk97TW6lv6odZY+kAD5x5feZj9aQvTFjMOnuA0/iIS8kE2JIRrsgmfQKrIRGxXy1jSfvnqpkYZ3aWT1Pmk1F8Qu7DY2hHk6OdTyJPd54vT24OrkraybN88R3gZsh4ijvtAZ6XOj2lLdqApOS+quX4dcUv521UkeGU38s2PrChzCd3y3xunluW+YSnNMT9W798204Le1u1+oua/g4X8StKnyV6QmWqIK6X1KA3tIMusugrfaef9Gvxb7FcrBbXU9PCQoZ5RBee4vN/3q09Hg==

�3(f(j), x)

AAAGenicpVTLbtNQEJ0GGkp5pSCxYWOlqpSobXCqIrqMYMOCRUFNW6kpke3eNCZ+4Qc0CvkLvoAFG/ih/gH8AwvOjJ0mbRK3Er6KM54758yd1zUDx45iXT9fKNy6vVi8s3R3+d79Bw8flVYe70d+ElqqafmOHx6aRqQc21PN2I4ddRiEynBNRx2Yvde8f/BZhZHte3txP1DHrnHq2R3bMmKo2qWnraBrt7crnQ+DysfqcKNluoOzYbVdWtVrujzatFDPhNVGubX+7bzR3/VXCj+oRSfkk0UJuaTIoxiyQwZFWEdUJ50C6I5pAF0IyZZ9RUNaBjaBlYKFAW0P71N8HWVaD9/MGQnaghcHvxBIjdaA8WEXQmZvmuwnwszaedwD4eSz9fFvZlwutDF1ob0ON7K8Oc6EVX6sMXVoR2K0EXMgGo7eyrwkkjWOTJuIOgZDAB3LJ9gPIVuCHNVBE0wkueHcG7L/WyxZy99WZpvQn9wo2KcjcfxPJOzTkSp+gZzabwKhTdjP599DNzF3B3jvEnO61oQzkjztgNcEK2eBcdoFKi/KLroizSnnqDPBPfYwtnHk1L0b5U2bs5gxAo8LZFrhJr2nt1Lf1A+zxtIBHjjz+szH6klfmLCYdfYAp/ERl5IJsCUjXJFN+gRWQyJiv1rGkvbPdTMxzuwsn6bMJ6P4hNyHJ9COJke7mESe7jxfntwcXJW0k2f54jvAzZDxFHfaAzwvdXpOW7QBScl9Vcvx64pfztuoIsMpv1dtfGBDmU/ulvncPLcs8wnPaIj7t371tp0W9rdq9Rc1/R0u4leUPkv0jMpUQVwvqUFvaBddZNFX+k4/6dfi32K5WC2up6aFhQzzhC49xe1/5Lw9Hw==

�4(f(j), x)

AAAGe3icpVTNThNRFD6gVMQfii5caMyNDYkxUKcobd0RjYkLF2AokAAhM9PbMun8OXMrENKlT2LiSve8im+g8RlM/M6ZKS1CC4lz0+mZc8/3nXv+rhP7Xmos68fE5LXrU4Ub0zdnbt2+c3e2OHdvI426iasbbuRHyZZjp9r3Qt0wnvH1VpxoO3B8vel03vD+5iedpF4UrpujWO8Gdjv0Wp5rG6j2ig92jD40x2+bba0OtNfeNypqqZ61VyxZZaterb54paxytfbSqlUh1JdrNaumKmVLntLKo5O1358fn6xGc5NfaYeaFJFLXQpIU0gGsk82pVjbVCGLYuh26Ri6BJIn+5p6NANsF1YaFja0Hbzb+NrOtSG+mTMVtAsvPn4JkIrmgYlgl0Bmb0r2u8LM2lHcx8LJZzvCv5NzBdAa2of2Mlzf8uo4B1bjYzXUorrE6CHmWDQcvZt76UrWODI1FLUBQwwdy03sJ5BdQfbroASTSm4497bs/xRL1vK3m9t26dfYKNinL3H8TyTs05cqHkDO7BeBUEP2o/nX0U3M3QI+PMOcrXnhTCVPdfA6YOUsME6dosZFuY+uyHLKOWoNcQ88DGx8OXXnSnlTIxYzpuAJgMwq3KAP9F7qm/lhViMdEIJzXJ9FWB3pCwcWF509xmkixKVlAjzJCFdkkT6C1ZaI2K/KWbL+uWwmBpm9yKcj88koPiH3YRPa/uSo00nk6R7nK5Sbg6uSdfJFvvgOCHKkOced9QDPS4We0xItQNJyX5XH+A3EL+etX5HeOb//2kTAJjKf3C2juXluWeYTHlIP92//klWjhY2lcmW5bK3hIn5N2TNND+kJPUVcNVqhd7SKLuIMfaFv9H3qT6FUeFZYyEwnJ3LMfTrzFJb/Ak7YPuE=

Edge weight of 0

AAAC3HicjVHLSsNAFD2Nr1pfVcGNm2ARXJVECrosdeOygrWFtpRkOq2haRImE7HUrlyKWz/Arf6P+Af6F94ZU1CL6IQkZ86958zce93I92JpWa8ZY25+YXEpu5xbWV1b38hvbl3EYSIYr7HQD0XDdWLuewGvSU/6vBEJ7gxdn9fdwYmK16+4iL0wOJejiLeHTj/weh5zJFGd/E5L8ms5roxMFgaxFAlT/KSTL1hFSy9zFtgpKCBd1TD/gha6CMGQYAiOAJKwDwcxPU3YsBAR18aYOEHI03GOCXKkTSiLU4ZD7IC+fdo1UzagvfKMtZrRKT69gpQm9kkTUp4grE4zdTzRzor9zXusPdXdRvR3U68hsRKXxP6lm2b+V6dqkejhWNfgUU2RZlR1LHVJdFfUzc0vVUlyiIhTuEtxQZhp5bTPptbEunbVW0fH33SmYtWepbkJ3tUtacD2z3HOgvph0S4VbfusVChX0llnsYs9HNBAj1DGKaqokfkNHvGEZ6Nt3Bp3xv1nqpFJNdv4toyHD6ZymWA=

By constructionAAAGlXicpVTLbtNQEJ0GMKW8UliwYGO1qpSKNtgFRDeVIkCIBRIFNW2lpkS2c9OY+IXtQKMof4HEH/AXbOEb+gfwF5wZO03aPFoJX8UZz51z5s7r2pHnJqlhnMwVrly9pl2fv7Fw89btO3eLi/d2k7ATO6rqhF4Y79tWojw3UNXUTT21H8XK8m1P7dntl7y/90XFiRsGO2k3Uoe+dRS4TdexUqjqRbMWtdz6k1LzY6/0abW/ptdsv3fcX90S/dNxvVEvLhtlQx59XDBzYbmyVHv07aTS3Q4XCz+oRg0KyaEO+aQooBSyRxYlWAdkkkERdIfUgy6G5Mq+oj4tANuBlYKFBW0b7yN8HeTaAN/MmQjagRcPvxhInVaACWEXQ2Zvuux3hJm107h7wsln6+Lfzrl8aFNqQXsRbmB5eZwNq9mxptSkTYnRRcyRaDh6J/fSkaxxZPpI1CkYIuhYbmA/huwIclAHXTCJ5IZzb8n+H7FkLX87uW2H/s6Mgn16Esf/RMI+PaniV8iZ/ToQ+oj9dP4ddBNzN4EPzjBna0U4E8nTJnhtsHIWGKefomZF2UJXZDnlHDVHuIcehjaenLp9qbzpUxYzJuDxgcwqXKUP9Fbqm/lh1lQ6IADnrD4LsdrSFzYsJp09wmlCxKVkAlzJCFdknT6D1ZKI2K+es2T9c9FMDDM7yact88koPiH3YQPaweTop5PI0z3LVyA3B1cl6+RJvvgO8HNkOsad9QDPi0mPaYPWICm5r8oz/Pril/M2qEh/zO95mxDYWOaTu2U6N88ty3zCY+rj/jXP37bjwu5G2XxWNt7jIn5B2TNPD2mJSojrOVXoDW2jixz6Tj/pF/3WHmhb2ivtdWZamMsx9+nMo737B06xRdY=

�3(f(j), x) = �4(f

(j), x) = 0

AAAGg3icpVRbTxNREB5QKuKt6KNGTyQkGKHutgLlwYRoYnzQBJRbwi2721NYuzf3opCmj/wVX3zVhL/iP9D4J/xmdstFaCFxT7qdnTPfN2dux448N0kN4+fA4JWrQ6Vrw9dHbty8dftOefTuShJmsaOXndAL4zXbSrTnBno5dVNPr0Wxtnzb06t26xXvr37WceKGwVK6H+lN39oJ3KbrWClU2+VHG6neS9sfdq3I0/vqXdjIPD2pOur1Vnvi45POdnnMqMzNGXVzRhmV6eqsWTMhGLVabaauzIohz9j8g8PFPwcPDxfC0cGvtEENCsmhjHzSFFAK2SOLEqx1MsmgCLpNakMXQ3JlX1OHRoDNYKVhYUHbwnsHX+uFNsA3cyaCduDFwy8GUtE4MCHsYsjsTcl+Jsys7cXdFk4+2z7+7YLLhzalXWgvwnUtL4+zYdU/1pSaVJcYXcQciYajdwovmWSNI1Mnok7BEEHHcgP7MWRHkN06KMEkkhvOvSX7v8SStfztFLYZ/e4bBfv0JI7/iYR9elLFL5Bz+ykg1An73vxL6CbmbgIfnGLO17hwJpKnOnhtsHIWGKeOUP2i3EVX5DnlHDVPcB97OLbx5NStS+VN9VjMmIDHBzKv8DK9p7dS39wPs6bSAQE4+/VZiNWSvrBhcd7ZI5wmRFxaJsCVjHBFpugTWC2JiP2qgiXvn4tm4jiz5/m0ZT4ZxSfkPmxA250cdTSJPN39fAVyc3BV8k4+zxffAX6BTM9w5z3A82LSM6rSJCQt91Wlj19f/HLeuhXpnPH7r00IbCzzyd3Sm5vnlmU+4R7x/du9ZFVvYaVaMacrxiIu4peUP8N0nx7TBOKapXl6QwvoIocO6Bt9px+lodLTUrX0PDcdHCgw9+jUU3rxF7YxQNo=

Shapley Module, F (j)

AAAGdHicpVTLbtNQEJ0GGkp5pbCEhdUoUiraYEdCdBnBhgWLgpq2Ulsi27lp3fiF7ZREUf6CJVv6T/0D+AHWnBk7TdokbiV8FWc8d86ZO69rha4TJ7p+uVS4d3+5+GDl4eqjx0+ePiutPd+Lg15kq6YduEF0YJmxch1fNRMncdVBGCnTs1y1b3U/8P7+uYpiJ/B3k0Gojj3zxHc6jm0mULVKpc7XYfVsY1Ttt4zNfqu+0SqV9ZoujzYrGJlQbqwfvf5x2RjsBGuFCzqiNgVkU488UuRTAtklk2KsQzJIpxC6YxpCF0FyZF/RiFaB7cFKwcKEtov3Cb4OM62Pb+aMBW3Di4tfBKRGFWAC2EWQ2Zsm+z1hZu0i7qFw8tkG+LcyLg/ahE6hvQ03trw7zoJVfqwJdWhbYnQQcygajt7OvPQkaxyZNhV1AoYQOpbb2I8g24Ic10ETTCy54dybsv9bLFnL33Zm26M/uVGwT1fi+J9I2KcrVfwOObXfAkKbsl/Mv4tuYu4O8P415nRVhDOWPG2D1wIrZ4Fx2hUqL8pTdEWaU85RZ4p74mFi48qpu3fKm7ZgMWMMHg/ItMJN+kKfpL6pH2ZNpAN8cOb1WYDVlb6wYDHv7CFOEyAuJRPgSEa4Ilv0DaymRMR+tYwl7Z/bZmKS2Xk+LZlPRvEJuQ/b0I4nR7uaRJ7uPF++3BxclbST5/niO8DLkMkMd9oDPC8GvaE6bUJScl/Vcvx64pfzNq7IaMbvTZsA2Ejmk7tlMTfPLct8wj6NcP8aN2/bWWGvXjPe1vTPuIjfU/qs0EtapyriekcN+kg76CKbzukn/aKL5b/FV8VysZKaFpYyzAu69hRr/wBowjop

f (j)(x1, x2)

AAAGdHicpVTLTttQEB3SklL6Cu2yXVhEkYIKqR2pKsuo3XTRBa0IIAGNbOcGTPyqH5Qoyl902W35J/6g/YGue2bsECCJQaqv4oznzjlz53Wt0HXiRNcvFkr37i+WHyw9XH70+MnTZ5WV5ztxkEa2atuBG0R7lhkr1/FVO3ESV+2FkTI9y1W7Vv8D7++eqih2An87GYTq0DOPfKfn2GYCVadS6X0d1k/WRvWzjrEedZprnUpVb+jyaNOCkQvV1urB6x8XrcFWsFI6pwPqUkA2peSRIp8SyC6ZFGPtk0E6hdAd0hC6CJIj+4pGtAxsCisFCxPaPt5H+NrPtT6+mTMWtA0vLn4RkBrVgAlgF0Fmb5rsp8LM2nncQ+Hksw3wb+VcHrQJHUN7G25seXecBaviWBPq0abE6CDmUDQcvZ17SSVrHJl2JeoEDCF0LHexH0G2BTmugyaYWHLDuTdl/7dYspa/7dw2pT+FUbBPV+L4n0jYpytV/A45s98AQrtiP59/G93E3D3g/WvM2aoJZyx52gSvBVbOAuO0S1RRlMfoiiynnKPeFe6Jh4mNK6fu3ylv2pzFjDF4PCCzCrfpC32S+mZ+mDWRDvDBWdRnAVZf+sKCxayzhzhNgLiUTIAjGeGKbNA3sJoSEfvVcpasf26biUlmZ/m0ZD4ZxSfkPuxCO54c7XISebqLfPlyc3BVsk6e5YvvAC9HJlPcWQ/wvBj0hpq0DknJfdUo8OuJX87buCKjKb83bQJgI5lP7pb53Dy3LPMJz2iE+9e4edtOCzvNhvG2oX/GRfyesmeJXtIq1RHXO2rRR9pCF9l0Sj/pF50v/i2/KlfLtcy0tJBjXtC1p9z4B0S2OiM=

f (j)(x1, r2)

AAAGdHicpVTLbtNQEJ0GGkp5pbCEhdUoUiraYEdCdBnBhgWLgpq2Ulsi27lpTfzCj0IU5S9YsqX/1D+AH2DNmbHTpE3iVsJXccZz55y587pW6DpxousXS6U7d5fL91burz54+Ojxk8ra0704SCNbte3ADaIDy4yV6/iqnTiJqw7CSJme5ap9q/+O9/fPVBQ7gb+bDEJ17JknvtNzbDOBqlOp9D4P6182RvWoY2xGneZGp1LVG7o82qxg5EK1tX708sdFa7ATrJXO6Yi6FJBNKXmkyKcEsksmxViHZJBOIXTHNIQuguTIvqIRrQKbwkrBwoS2j/cJvg5zrY9v5owFbcOLi18EpEY1YALYRZDZmyb7qTCzdhH3UDj5bAP8WzmXB21Cp9DehBtb3h5nwao41oR6tC0xOog5FA1Hb+deUskaR6ZNRZ2AIYSO5S72I8i2IMd10AQTS24496bs/xZL1vK3ndum9KcwCvbpShz/Ewn7dKWK3yBn9ltAaFP2i/l30U3M3QPev8KcrZpwxpKnbfBaYOUsME67RBVFeYquyHLKOepNcU88TGxcOXX/VnnTFixmjMHjAZlVuE2f6IPUN/PDrIl0gA/Ooj4LsPrSFxYs5p09xGkCxKVkAhzJCFdki76C1ZSI2K+Ws2T9c9NMTDI7z6cl88koPiH3YRfa8eRol5PI013ky5ebg6uSdfI8X3wHeDkymeHOeoDnxaBX1KRNSEruq0aBX0/8ct7GFRnN+L1uEwAbyXxytyzm5rllmU/4nUa4f43rt+2ssNdsGK8b+kdcxG8pe1boOa1THXG9oRa9px10kU1n9JN+0fny3/KLcrVcy0xLSznmGV15yo1/IJI6HQ==

f (j)(r1, r2)

AAAGdHicpVTLTttQEB3SklL6Cu2yXVhEkYIKqR2pKsuo3XTRBa0IIAGNbOcGTPyqH5Qoyl902W35J/6g/YGue2bsECCJQaqv4oznzjlz53Wt0HXiRNcvFkr37i+WHyw9XH70+MnTZ5WV5ztxkEa2atuBG0R7lhkr1/FVO3ESV+2FkTI9y1W7Vv8D7++eqih2An87GYTq0DOPfKfn2GYCVadS6X0d1k/WRvWoY6yfdZprnUpVb+jyaNOCkQvV1urB6x8XrcFWsFI6pwPqUkA2peSRIp8SyC6ZFGPtk0E6hdAd0hC6CJIj+4pGtAxsCisFCxPaPt5H+NrPtT6+mTMWtA0vLn4RkBrVgAlgF0Fmb5rsp8LM2nncQ+Hksw3wb+VcHrQJHUN7G25seXecBaviWBPq0abE6CDmUDQcvZ17SSVrHJl2JeoEDCF0LHexH0G2BTmugyaYWHLDuTdl/7dYspa/7dw2pT+FUbBPV+L4n0jYpytV/A45s98AQrtiP59/G93E3D3g/WvM2aoJZyx52gSvBVbOAuO0S1RRlMfoiiynnKPeFe6Jh4mNK6fu3ylv2pzFjDF4PCCzCrfpC32S+mZ+mDWRDvDBWdRnAVZf+sKCxayzhzhNgLiUTIAjGeGKbNA3sJoSEfvVcpasf26biUlmZ/m0ZD4ZxSfkPuxCO54c7XISebqLfPlyc3BVsk6e5YvvAC9HJlPcWQ/wvBj0hpq0DknJfdUo8OuJX87buCKjKb83bQJgI5lP7pb53Dy3LPMJz2iE+9e4edtOCzvNhvG2oX/GRfyesmeJXtIq1RHXO2rRR9pCF9l0Sj/pF50v/i2/KlfLtcy0tJBjXtC1p9z4B0SeOiM=

f (j)(r1, x2)

AAAGjXicpVTbThRBEC1QV8QLoI8a05GQaILr7K4ESIwh+oAPxoByS1hCZoZedrI9F2d6BbLZR5/9C//AhCf9Dv9A4094qmaWi7ALidNhqa6uc6rr1l5igsw6zs+h4StXr5Wuj9wYvXnr9p2x8Ym7a1ncTn296scmTjc8N9MmiPSqDazRG0mq3dAzet1rvebz9U86zYI4WrEHid4K3d0oaAS+a6HaHn9St3rfdj40XWPiPdWpZ77CJnmnbXdadeuha5u+azqL3e3xSaf8vDZXna0qp+zIB2F+vlar1FSl0EwuPDhc/vP54eFSPDH8leq0QzH51KaQNEVkIRtyKcPapAo5lEC3RR3oUkiBnGvq0iiwbVhpWLjQtvC7i91moY2wZ85M0D68GPylQCqaAiaGXQqZvSk5bwsza/txd4ST73aA/17BFUJrqQntRbie5eVxHqwGx2qpQXMSY4CYE9Fw9H7hpS1Z48jUiagtGBLoWN7BeQrZF2SvDkowmeSGc+/K+S+xZC3v/cK2Tb8HRsE+jcTxP5GwTyNV3IOc2z8FQp2w78+/gm5i7gbw0SnmfE0JZyZ5mgOvB1bOAuPUEWpQlE10RZ5TzlHjBPexh2MbI7duXSpvqs9ixgw8IZB5hVfpPb2V+uZ+mNVKB0TgHNRnMVZL+sKDxXl3T3CbGHFpmYBAMsIVeUofwepKROxXFSx5/1w0E8eZPc+nJ/PJKL4h9+EOtL3JUUeTyNM9yFckLwdXJe/k83zxGxAWSHuGO+8BnpcKPaMqTUPS8l6VB/gNxS/nrVeR7hm//9rEwKYyn9wt/bl5blnmG+4Tv7+9R1b1F9aq5cpM2VnGQ/yK8m+E7tMjeoy4ZmmB3tASusinL/SNvtOP0lhppvSi9DI3HR4qMPfo1Fda/Av5y0XC

Shallow ShapNet, G

AAAGdHicpVTLbtNQEJ0GGkp5pbCEhdUoUira1I6E6DICCbFgUVDTVmpLZDs3rRW/8KMkivIXLNnSf+ofwA+w5szYadLm0Ur4Ks547pwzd17XCl0nTnT9cqlw7/5y8cHKw9VHj588fVZae74fB2lkq6YduEF0aJmxch1fNRMncdVhGCnTs1x1YHXf8/7BuYpiJ/D3kn6oTjzz1Hc6jm0mULVKpQ9fB1VjY1jttYzNXqu+0SqV9ZoujzYtGLlQbqwfv/5x2ejvBmuFCzqmNgVkU0oeKfIpgeySSTHWERmkUwjdCQ2giyA5sq9oSKvAprBSsDCh7eJ9iq+jXOvjmzljQdvw4uIXAalRBZgAdhFk9qbJfirMrJ3HPRBOPlsf/1bO5UGb0Bm0t+FGlnfHWbBaHGtCHdqRGB3EHIqGo7dzL6lkjSPTJqJOwBBCx3Ib+xFkW5CjOmiCiSU3nHtT9n+LJWv5285tU/qzMAr26Uoc/xMJ+3Slit8hZ/ZbQGgT9vP599BNzN0B3r/GnK2KcMaSpx3wWmDlLDBOu0ItivIMXZHllHPUmeAeexjbuHLq7p3yps1ZzBiDxwMyq3CTvtAnqW/mh1kT6QAfnIv6LMDqSl9YsJh19hCnCRCXkglwJCNckS36BlZTImK/Ws6S9c9tMzHO7Cyflswno/iE3IdtaEeTo11NIk/3Il++3BxclayTZ/niO8DLkckUd9YDPC8GbVOdNiEpua9qC/x64pfzNqrIcMrvTZsA2Ejmk7tlPjfPLct8wh4Ncf8aN2/baWG/XjPe1PTPuIjfUfas0EtapyrieksN+ki76CKbzukn/aKL5b/FV8VysZKZFpZyzAu69hRr/wBOqjnQ

F (1)(x1, x2)

AAAGdHicpVTLbtNQEJ0GGkp5pbCEhdUoUiraEAchuoxAQixYFNS0ldoS2c5Na8Uv/CiJovwFS7b0n/oH8AOsOTN2mrRJ3Er4Ks547pwzd17XDBw7iuv1i6XCnbvLxXsr91cfPHz0+Elp7ele5CehpVqW7/jhgWlEyrE91Yrt2FEHQagM13TUvtl7z/v7ZyqMbN/bjQeBOnaNE8/u2pYRQ9UulT58HVYbG6Nqv61v9tuvN9qlcr1Wl0ebFfRMKDfXj17+uGgOdvy1wjkdUYd8sighlxR5FEN2yKAI65B0qlMA3TENoQsh2bKvaESrwCawUrAwoO3hfYKvw0zr4Zs5I0Fb8OLgFwKpUQUYH3YhZPamyX4izKxdxD0UTj7bAP9mxuVCG9MptDfhxpa3x5mwyo81pi5tS4w2Yg5Ew9FbmZdEssaRaVNRx2AIoGO5g/0QsiXIcR00wUSSG869Ifu/xZK1/G1ltgn9yY2CfToSx/9Ewj4dqeJ3yKn9FhDalP1i/l10E3N3gfeuMKerIpyR5GkbvCZYOQuM0y5ReVGeoivSnHKOulPcEw8TG0dO3btV3rQFixkj8LhAphVu0Rf6JPVN/TBrLB3ggTOvz3ysnvSFCYt5Zw9wGh9xKZkAWzLCFdmib2A1JCL2q2Usaf/cNBOTzM7zacp8MopPyH3YgXY8OdrlJPJ05/ny5ObgqqSdPM8X3wFuhoxnuNMe4HnR6RU1aBOSkvuqluPXFb+ct3FFRjN+r9v4wIYyn9wti7l5blnmE/ZphPtXv37bzgp7jZr+plb/jIv4HaXPCj2ndaoirrfUpI+0gy6y6Ix+0i86X/5bfFEsFyupaWEpwzyjK0+x9g9atDnS

F (2)(x1, x3)

AAAGdHicpVTLbtNQEJ0GGkp5pbCEhdUoUira1C4guoxAQixYFNS0ldoS2c5Na8UvbKckivIXLNnSf+ofwA+w5szYadImcSvhqzjjuXPO3HldK3SdONH1i4XCnbuLxXtL95cfPHz0+Elp5eleHHQjWzXswA2iA8uMlev4qpE4iasOwkiZnuWqfavznvf3z1QUO4G/m/RDdeyZJ77TdmwzgapZKn34Oqi+WhtWe01jvdd8vdYslfWaLo82LRiZUK6vHr38cVHv7wQrhXM6ohYFZFOXPFLkUwLZJZNirEMySKcQumMaQBdBcmRf0ZCWge3CSsHChLaD9wm+DjOtj2/mjAVtw4uLXwSkRhVgAthFkNmbJvtdYWbtPO6BcPLZ+vi3Mi4P2oROob0JN7K8Pc6CVX6sCbVpW2J0EHMoGo7ezrx0JWscmTYRdQKGEDqWW9iPINuCHNVBE0wsueHcm7L/WyxZy992ZtulP7lRsE9X4vifSNinK1X8Djm13wBCm7Cfz7+LbmLuNvD+FeZ0VYQzljxtg9cCK2eBcdolKi/KU3RFmlPOUXuCe+xhbOPKqTu3yps2ZzFjDB4PyLTCDfpCn6S+qR9mTaQDfHDm9VmA1ZG+sGAx6+whThMgLiUT4EhGuCIb9A2spkTEfrWMJe2fm2ZinNlZPi2ZT0bxCbkPW9COJke7nESe7jxfvtwcXJW0k2f54jvAy5DJFHfaAzwvBm3SFq1DUnJf1XL8euKX8zaqyHDK73WbANhI5pO7ZT43zy3LfMIeDXH/Gtdv22lhb6tmvKnpn3ERv6P0WaLntEpVxPWW6vSRdtBFNp3RT/pF54t/iy+K5WIlNS0sZJhndOUp1v4BZr451A==

F (3)(x1, x4)

AAAGdHicpVRLTxNRFD6gVMRX0aUuJjRNSoQyrRpZNpoYFy7QUCABbGamtzDpvJwHtmn6L1y6lf/EP9A/4NrvnJnSQtuBxLnp9My55/vOPa9rBo4dxbp+sbB45+5S4d7y/ZUHDx89flJcfboX+UloqablO354YBqRcmxPNWM7dtRBECrDNR21b3bf8/7+mQoj2/d2436gjl3jxLM7tmXEULWKxQ9fB5XX68NKr1Xf6LVerbeKJb2qy6NNC7VMKDXWjl7+uGj0d/zVxXM6ojb5ZFFCLinyKIbskEER1iHVSKcAumMaQBdCsmVf0ZBWgE1gpWBhQNvF+wRfh5nWwzdzRoK24MXBLwRSozIwPuxCyOxNk/1EmFk7j3sgnHy2Pv7NjMuFNqZTaG/CjSxvjzNhlR9rTB3alhhtxByIhqO3Mi+JZI0j0yaijsEQQMdyG/shZEuQozpogokkN5x7Q/Z/iyVr+dvKbBP6kxsF+3Qkjv+JhH06UsXvkFP7TSC0Cfv5/LvoJubuAO9dYU5XWTgjydM2eE2wchYYp12i8qI8RVekOeUcdSa4xx7GNo6cunurvGlzFjNG4HGBTCvcpC/0Seqb+mHWWDrAA2den/lYXekLExazzh7gND7iUjIBtmSEK7JJ38BqSETsV8tY0v65aSbGmZ3l05T5ZBSfkPuwDe1ocrTLSeTpzvPlyc3BVUk7eZYvvgPcDBlPcac9wPNSoy2q0wYkJfdVNcevK345b6OKDKf8XrfxgQ1lPrlb5nPz3LLMJ+zREPdv7fptOy3s1au1N1X9My7id5Q+y/Sc1qiCuN5Sgz7SDrrIojP6Sb/ofOlv4UWhVCinposLGeYZXXkK1X9szDnV

F (4)(x2, x3)

AAAGdHicpVRLTxNRFD5UqYivoktdTGialAhl2khk2WhiXLhAQ4EEsJkZbmHSeTkPbNP0X7h0K/+Jf6B/wLXfOTOlhbYDiXPT6Zlzz/ede17XDBw7inX9cqFw7/5i8cHSw+VHj588fVZaeb4X+UloqZblO354YBqRcmxPtWI7dtRBECrDNR21b3bf8/7+uQoj2/d2436gjl3j1LM7tmXEULVLpQ9fB9WttWG1126s99pv1tqlsl7T5dGmhXomlJurR69/XDb7O/5K4YKO6IR8sighlxR5FEN2yKAI65DqpFMA3TENoAsh2bKvaEjLwCawUrAwoO3ifYqvw0zr4Zs5I0Fb8OLgFwKpUQUYH3YhZPamyX4izKydxz0QTj5bH/9mxuVCG9MZtLfhRpZ3x5mwyo81pg5tS4w2Yg5Ew9FbmZdEssaRaRNRx2AIoGP5BPshZEuQozpogokkN5x7Q/Z/iyVr+dvKbBP6kxsF+3Qkjv+JhH06UsXvkFP7DSC0Cfv5/LvoJubuAO9dY05XRTgjydM2eE2wchYYp12h8qI8Q1ekOeUcdSa4xx7GNo6cununvGlzFjNG4HGBTCvcoi/0Seqb+mHWWDrAA2den/lYXekLExazzh7gND7iUjIBtmSEK7JB38BqSETsV8tY0v65bSbGmZ3l05T5ZBSfkPvwBNrR5GhXk8jTnefLk5uDq5J28ixffAe4GTKe4k57gOelTpvUoHVISu6rWo5fV/xy3kYVGU75vWnjAxvKfHK3zOfmuWWZT9ijIe7f+s3bdlrYa9TqWzX9My7id5Q+S/SSVqmKuN5Skz7SDrrIonP6Sb/oYvFv8VWxXKykpoWFDPOCrj3F2j941jnX

F (5)(x2, x4)

AAAGdHicpVTLbtNQEJ0GGkp5pbCEhdUoUira4JRXlxFIiAWLgpq2Ulsi27lprfiFHyVRlL9gyZb+U/8AfoA1Z8ZOkzaJWwlfxRnPnXPmzuuagWNHsa6fLxRu3V4s3lm6u3zv/oOHj0orj3cjPwkt1bR8xw/3TSNSju2pZmzHjtoPQmW4pqP2zO573t87VWFk+95O3A/UkWsce3bHtowYqlap9OHroPpmbVjttV6u91qv1lqlsl7T5dGmhXomlBurh89/nDf62/5K4YwOqU0+WZSQS4o8iiE7ZFCEdUB10imA7ogG0IWQbNlXNKRlYBNYKVgY0HbxPsbXQab18M2ckaAteHHwC4HUqAKMD7sQMnvTZD8RZtbO4x4IJ5+tj38z43KhjekE2utwI8ub40xY5ccaU4e2JEYbMQei4eitzEsiWePItImoYzAE0LHcxn4I2RLkqA6aYCLJDefekP3fYsla/rYy24T+5EbBPh2J438iYZ+OVPE75NR+Awhtwn4+/w66ibk7wHuXmNNVEc5I8rQFXhOsnAXGaReovChP0BVpTjlHnQnusYexjSOn7t4ob9qcxYwReFwg0wo36Qt9kvqmfpg1lg7wwJnXZz5WV/rChMWsswc4jY+4lEyALRnhimzQN7AaEhH71TKWtH+um4lxZmf5NGU+GcUn5D5sQzuaHO1iEnm683x5cnNwVdJOnuWL7wA3Q8ZT3GkP8LzU6QVt0jokJfdVLcevK345b6OKDKf8XrXxgQ1lPrlb5nPz3LLMJ+zREPdv/eptOy3sbtbqr2v6Z1zE7yh9lugprVIVcb2lBn2kbXSRRaf0k37R2eLf4rNiuVhJTQsLGeYJXXqKtX+E5DnZ

F (6)(x3, x4)

AAAGe3icpVTLbtNQEJ0GGkp5pbBgwcZqVCmFNNiVKrqMQEIsWBTUtJWaEtnOTWPFL/yARlH+gj9AYgUf1D+Aj0DizNhp0ubRSvgqznjunDN3XtcKXSdOdP18qXDr9nLxzsrd1Xv3Hzx8VFp7fBAHaWSrhh24QXRkmbFyHV81Eidx1VEYKdOzXHVo9d7w/uEXFcVO4O8n/VCdeOap73Qc20ygapWeNsOu0zIqbz8NKsbmsKo1LW9wNtxslcp6TZdHmxaMXCjX15svvp3X+3vBWuEHNalNAdmUkkeKfEogu2RSjHVMBukUQndCA+giSI7sKxrSKrAprBQsTGh7eJ/i6zjX+vhmzljQNry4+EVAarQBTAC7CDJ702Q/FWbWzuMeCCefrY9/K+fyoE2oC+11uJHlzXEWrBbHmlCHdiVGBzGHouHo7dxLKlnjyLSJqBMwhNCx3MZ+BNkW5KgOmmBiyQ3n3pT932LJWv62c9uU/iyMgn26Esf/RMI+XaniV8iZ/RYQ2oT9fP59dBNzd4D3LzFna0M4Y8nTLngtsHIWGKddoBZF2UVXZDnlHHUmuMcexjaunLp3o7xpcxYzxuDxgMwq3KCP9F7qm/lh1kQ6wAfnoj4LsHrSFxYsZp09xGkCxKVkAhzJCFdkiz6D1ZSI2K+Ws2T9c91MjDM7y6cl88koPiH3YRva0eRoF5PI073Ily83B1cl6+RZvvgO8HJkMsWd9QDPi0EvaZuqkJTcV7UFfj3xy3kbVWQ45feqTQBsJPPJ3TKfm+eWZT7hGQ1x/xpXb9tp4WC7ZuzU9A+4iF9T9qzQM1qnCuJ6RXV6R3voIs7Qd/pJv5b/FsvF58VqZlpYyjFP6NJT3PkHp+Y87Q==

�1(F(1), x)

AAAGe3icpVTLbtNQEJ0WGkp5pbBgwcZqVCmFNNiRKrqMQEIsWBTUtJWaEtnOTWPFL/yARlH+gj9AYgUf1D+Aj0DizNhp0ubRSvgqznjunDN3XtcKXSdOdP18afnW7ZXCndW7a/fuP3j4qLj++CAO0shWDTtwg+jIMmPlOr5qJE7iqqMwUqZnuerQ6r3h/cMvKoqdwN9P+qE68cxT3+k4tplA1So+bYZdp1Urv/00KBtbw4rWtLzB2XCrVSzpVV0ebVowcqFU32i++HZe7+8F68s/qEltCsimlDxS5FMC2SWTYqxjMkinELoTGkAXQXJkX9GQ1oBNYaVgYULbw/sUX8e51sc3c8aCtuHFxS8CUqNNYALYRZDZmyb7qTCzdh73QDj5bH38WzmXB21CXWivw40sb46zYLU41oQ6tCsxOog5FA1Hb+deUskaR6ZNRJ2AIYSO5Tb2I8i2IEd10AQTS24496bs/xZL1vK3ndum9GdhFOzTlTj+JxL26UoVv0LO7LeB0Cbs5/Pvo5uYuwO8f4k5W5vCGUuedsFrgZWzwDjtArUoyi66Issp56gzwT32MLZx5dS9G+VNm7OYMQaPB2RW4QZ9pPdS38wPsybSAT44F/VZgNWTvrBgMevsIU4TIC4lE+BIRrgi2/QZrKZExH61nCXrn+tmYpzZWT4tmU9G8Qm5D9vQjiZHu5hEnu5Fvny5ObgqWSfP8sV3gJcjkynurAd4Xgx6STWqQFJyX1UX+PXEL+dtVJHhlN+rNgGwkcwnd8t8bp5blvmEZzTE/WtcvW2nhYNa1dip6h9wEb+m7FmlZ7RBZcT1iur0jvbQRZyh7/STfq38LZQKzwuVzHR5Kcc8oUtPYecfrfY87g==

�2(F(1), x)

AAAGe3icpVTLbtNQEJ0WGkp5pbBgwcZqVCmFNNiRKrqMQEIsWBTUtJWaEtnOTWPFL/yARlH+gj9AYgUf1D+Aj0DizNhp0ubRSvgqznjunDN3XtcKXSdOdP18afnW7ZXCndW7a/fuP3j4qLj++CAO0shWDTtwg+jIMmPlOr5qJE7iqqMwUqZnuerQ6r3h/cMvKoqdwN9P+qE68cxT3+k4tplA1So+bYZdp2WU334alGtbw4rWtLzB2XCrVSzpVV0ebVowcqFU32i++HZe7+8F68s/qEltCsimlDxS5FMC2SWTYqxjMkinELoTGkAXQXJkX9GQ1oBNYaVgYULbw/sUX8e51sc3c8aCtuHFxS8CUqNNYALYRZDZmyb7qTCzdh73QDj5bH38WzmXB21CXWivw40sb46zYLU41oQ6tCsxOog5FA1Hb+deUskaR6ZNRJ2AIYSO5Tb2I8i2IEd10AQTS24496bs/xZL1vK3ndum9GdhFOzTlTj+JxL26UoVv0LO7LeB0Cbs5/Pvo5uYuwO8f4k5W5vCGUuedsFrgZWzwDjtArUoyi66Issp56gzwT32MLZx5dS9G+VNm7OYMQaPB2RW4QZ9pPdS38wPsybSAT44F/VZgNWTvrBgMevsIU4TIC4lE+BIRrgi2/QZrKZExH61nCXrn+tmYpzZWT4tmU9G8Qm5D9vQjiZHu5hEnu5Fvny5ObgqWSfP8sV3gJcjkynurAd4Xgx6STWqQFJyX1UX+PXEL+dtVJHhlN+rNgGwkcwnd8t8bp5blvmEZzTE/WtcvW2nhYNa1dip6h9wEb+m7FmlZ7RBZcT1iur0jvbQRZyh7/STfq38LZQKzwuVzHR5Kcc8oUtPYecfrfA87g==

�1(F(2), x)

AAAGe3icpVTLbtNQEJ0GGkp5NIUFCzZWo0oppMEJqugyAgmxYFFQ01ZqSmQ7N40Vv7AdaBTlL/gDJFbwQf0D+AgkzoydJm0StxK+ijOeO+fMndc1A8eOYl0/X8rdur2cv7Nyd/Xe/QcP1wrrjw4ivx9aqmH5jh8emUakHNtTjdiOHXUUhMpwTUcdmr03vH/4RYWR7Xv78SBQJ65x6tkd2zJiqFqFJ82ga7delt5+GpZqW6Oy1jTd4dloq1Uo6hVdHm1WqKZCsb7RfP7tvD7Y89dzP6hJbfLJoj65pMijGLJDBkVYx1QlnQLoTmgIXQjJln1FI1oFtg8rBQsD2h7ep/g6TrUevpkzErQFLw5+IZAabQLjwy6EzN402e8LM2sXcQ+Fk882wL+ZcrnQxtSF9jrc2PLmOBNW2bHG1KFdidFGzIFoOHor9dKXrHFk2lTUMRgC6FhuYz+EbAlyXAdNMJHkhnNvyP5vsWQtf1upbZ/+ZEbBPh2J438iYZ+OVPEr5MR+Gwhtyn4x/z66ibk7wHuXmJO1KZyR5GkXvCZYOQuM0y5QWVF20RVJTjlHnSnuiYeJjSOn7t0ob9qCxYwReFwgkwo36CO9l/omfpg1lg7wwJnVZz5WT/rChMW8swc4jY+4lEyALRnhimzTZ7AaEhH71VKWpH+um4lJZuf5NGU+GcUn5D5sQzueHO1iEnm6s3x5cnNwVZJOnueL7wA3RcYz3EkP8LxU6QXVqAxJyX1VyfDril/O27gioxm/V218YEOZT+6Wxdw8tyzzCc9ohPu3evW2nRUOapXqTkX/gIv4NSXPCj2lDSohrldUp3e0hy7iDH2nn/Rr+W++mH+WLyemuaUU85guPfmdf7oQPPA=

�3(F(2), x)

AAAGe3icpVTdThNREB5QKuIPRS+88GZDQ1K01C6GyGWjifHCCzQUSCg2u9tTuun+uT9K0/QtfAMTr/SBeAN9CBO/md3SQtuFxD3pdnbOfN+c+Ttm4NhRXKudLyzeur1UuLN8d+Xe/QcPV4trjw4iPwkt1bB8xw+PTCNSju2pRmzHjjoKQmW4pqMOzd4b3j/8osLI9r39uB+oE9c49eyObRkxVK3ik2bQtVt6+e2nQfnl5rCiNU13cDbcbBVLtWpNHm1a0DOhVF9vPv92Xu/v+WuLP6hJbfLJooRcUuRRDNkhgyKsY9KpRgF0JzSALoRky76iIa0Am8BKwcKAtof3Kb6OM62Hb+aMBG3Bi4NfCKRGG8D4sAshszdN9hNhZu087oFw8tn6+DczLhfamLrQXocbWd4cZ8IqP9aYOrQrMdqIORANR29lXhLJGkemTUQdgyGAjuU29kPIliBHddAEE0luOPeG7P8WS9byt5XZJvQnNwr26Ugc/xMJ+3Skil8hp/ZbQGgT9vP599FNzN0B3rvEnK4N4YwkT7vgNcHKWWCcdoHKi7KLrkhzyjnqTHCPPYxtHDl170Z50+YsZozA4wKZVrhBH+m91Df1w6yxdIAHzrw+87F60hcmLGadPcBpfMSlZAJsyQhXZIs+g9WQiNivlrGk/XPdTIwzO8unKfPJKD4h92Eb2tHkaBeTyNOd58uTm4OrknbyLF98B7gZMp7iTnuA50WnF7RNFUhK7qtqjl9X/HLeRhUZTvm9auMDG8p8crfM5+a5ZZlPeEZD3L/61dt2WjjYruo71doHXMSvKX2W6SmtUxlxvaI6vaM9dBFn6Dv9pF9LfwulwrNCJTVdXMgwj+nSU9j5B7P6PO8=

�1(F(3), x)

AAAGe3icpVTLbtNQEJ0GGkp5pbBgwcZqVCmFNjiFii4jkBALFgU1baWmRLZz01jxCz+gUZS/4A+QWMEH9Q/gI5A4M3aatEncSvgqznjunDN3XtcMHDuKdf1soXDj5mLx1tLt5Tt3791/UFp5uB/5SWiphuU7fnhoGpFybE81Yjt21GEQKsM1HXVg9t7w/sEXFUa27+3F/UAdu8aJZ3dsy4ihapUeN4Ou3XpZeftpUHmxPtzQmqY7OB2ut0plvarLo00LtUwo11ebz76d1fu7/krhBzWpTT5ZlJBLijyKITtkUIR1RDXSKYDumAbQhZBs2Vc0pGVgE1gpWBjQ9vA+wddRpvXwzZyRoC14cfALgdRoDRgfdiFk9qbJfiLMrJ3HPRBOPlsf/2bG5UIbUxfaq3Ajy+vjTFjlxxpTh3YkRhsxB6Lh6K3MSyJZ48i0iahjMATQsdzGfgjZEuSoDppgIskN596Q/d9iyVr+tjLbhP7kRsE+HYnjfyJhn45U8Svk1H4TCG3Cfj7/HrqJuTvAexeY07UmnJHkaQe8Jlg5C4zTzlF5UXbRFWlOOUedCe6xh7GNI6fuXStv2pzFjBF4XCDTCjfoI72X+qZ+mDWWDvDAmddnPlZP+sKExayzBziNj7iUTIAtGeGKbNJnsBoSEfvVMpa0f66aiXFmZ/k0ZT4ZxSfkPmxDO5oc7XwSebrzfHlyc3BV0k6e5YvvADdDxlPcaQ/wvNToOW3RBiQl91U1x68rfjlvo4oMp/xetvGBDWU+uVvmc/PcsswnPKUh7t/a5dt2Wtjfqta2q/oHXMSvKX2W6AmtUgVxvaI6vaNddBFn6Dv9pF+Lf4vl4tPiRmpaWMgwj+jCU9z+B8YqPPI=

�4(F(3), x)

AAAGe3icpVTLbtNQEJ0GGkp5NIUFCzZWo0oppMGJqOgyAgmxYFFQ01ZqSmQ7N40Vv7AdaBTlL/gDJFbwQf0D+AgkzoydJm0StxK+ijOeO+fMndc1A8eOYl0/X8rdur2cv7Nyd/Xe/QcP1wrrjw4ivx9aqmH5jh8emUakHNtTjdiOHXUUhMpwTUcdmr03vH/4RYWR7Xv78SBQJ65x6tkd2zJiqFqFJ82ga7dqpbefhqWXW6Oy1jTd4dloq1Uo6hVdHm1WqKZCsb7RfP7tvD7Y89dzP6hJbfLJoj65pMijGLJDBkVYx1QlnQLoTmgIXQjJln1FI1oFtg8rBQsD2h7ep/g6TrUevpkzErQFLw5+IZAabQLjwy6EzN402e8LM2sXcQ+Fk882wL+ZcrnQxtSF9jrc2PLmOBNW2bHG1KFdidFGzIFoOHor9dKXrHFk2lTUMRgC6FhuYz+EbAlyXAdNMJHkhnNvyP5vsWQtf1upbZ/+ZEbBPh2J438iYZ+OVPEr5MR+Gwhtyn4x/z66ibk7wHuXmJO1KZyR5GkXvCZYOQuM0y5QWVF20RVJTjlHnSnuiYeJjSOn7t0ob9qCxYwReFwgkwo36CO9l/omfpg1lg7wwJnVZz5WT/rChMW8swc4jY+4lEyALRnhimzTZ7AaEhH71VKWpH+um4lJZuf5NGU+GcUn5D5sQzueHO1iEnm6s3x5cnNwVZJOnueL7wA3RcYz3EkP8LxU6QXVqAxJyX1VyfDril/O27gioxm/V218YEOZT+6Wxdw8tyzzCc9ohPu3evW2nRUOapXqTkX/gIv4NSXPCj2lDSohrldUp3e0hy7iDH2nn/Rr+W++mH+WLyemuaUU85guPfmdf8AUPPE=

�2(F(4), x)

AAAGe3icpVTLbtNQEJ0GGkp5pbBgwcZqVCmFNjiFii4jkBALFgU1baWmRLZz01jxCz+gUZS/4A+QWMEH9Q/gI5A4M3aatEncSvgqznjunDN3XtcMHDuKdf1soXDj5mLx1tLt5Tt3791/UFp5uB/5SWiphuU7fnhoGpFybE81Yjt21GEQKsM1HXVg9t7w/sEXFUa27+3F/UAdu8aJZ3dsy4ihapUeN4Ou3XpReftpUHm5PtzQmqY7OB2ut0plvarLo00LtUwo11ebz76d1fu7/krhBzWpTT5ZlJBLijyKITtkUIR1RDXSKYDumAbQhZBs2Vc0pGVgE1gpWBjQ9vA+wddRpvXwzZyRoC14cfALgdRoDRgfdiFk9qbJfiLMrJ3HPRBOPlsf/2bG5UIbUxfaq3Ajy+vjTFjlxxpTh3YkRhsxB6Lh6K3MSyJZ48i0iahjMATQsdzGfgjZEuSoDppgIskN596Q/d9iyVr+tjLbhP7kRsE+HYnjfyJhn45U8Svk1H4TCG3Cfj7/HrqJuTvAexeY07UmnJHkaQe8Jlg5C4zTzlF5UXbRFWlOOUedCe6xh7GNI6fuXStv2pzFjBF4XCDTCjfoI72X+qZ+mDWWDvDAmddnPlZP+sKExayzBziNj7iUTIAtGeGKbNJnsBoSEfvVMpa0f66aiXFmZ/k0ZT4ZxSfkPmxDO5oc7XwSebrzfHlyc3BV0k6e5YvvADdDxlPcaQ/wvNToOW3RBiQl91U1x68rfjlvo4oMp/xetvGBDWU+uVvmc/PcsswnPKUh7t/a5dt2Wtjfqta2q/oHXMSvKX2W6AmtUgVxvaI6vaNddBFn6Dv9pF+Lf4vl4tPiRmpaWMgwj+jCU9z+B8YkPPI=

�3(F(4), x)

AAAGe3icpVTdbhJREJ6ixVr/qF544c2mpAlVigsJsZdEE+OFF9WUtkmpZHc5lA375+6iJYS38A1MvNIH6hvoQ5j4zexSaIFtE/eEZXbOfN+c+Ttm4NhRrOvnK7lbt1fzd9burt+7/+Dho8LG44PIH4SWalq+44dHphEpx/ZUM7ZjRx0FoTJc01GHZv8N7x9+UWFk+95+PAzUiWucenbXtowYqnbhaSvo2e1a6e2nUam+PS5rLdMdnY2324WiXtHl0eaFaioUG5utF9/OG

Shapley Explanation Networks · 2020. 12. 11. · 2.2 Shallow SHAPNET To aid in our notation, we...

Documents

Transcript of Shapley Explanation Networks · 2020. 12. 11. · 2.2 Shallow SHAPNET To aid in our notation, we...