# Beyond Importance Scores: Interpreting Tabular ML by Visualizing Feature
Semantics

Amirata Ghorbani
Stanford University
&Dina Berenbaum11footnotemark: 1
Demystify AI
Maor Ivgi
Demystify AI
&Yuval Dafna
Demystify AI
&James Zou22footnotemark: 2
Stanford University
These authors contributed equally.Corresponding authors<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Interpretability is becoming an active research topic as machine learning (ML)
models are more widely used to make critical decisions. Tabular data is one of
the most commonly used modes of data in diverse applications such as
healthcare and finance. Much of the existing interpretability methods used for
tabular data only report _feature-importance_ scores—either locally (per
example) or globally (per model)—but they do not provide interpretation or
visualization of how the features interact. We address this limitation by
introducing Feature Vectors, a new global interpretability method designed for
tabular datasets. In addition to providing _feature-importance_ , Feature
Vectors discovers the inherent semantic relationship among features via an
intuitive feature visualization technique. Our systematic experiments
demonstrate the empirical utility of this new method by applying it to several
real-world datasets. We further provide an easy-to-use Python package for
Feature Vectors.

##  1 Introduction

As machine learning (ML) models have become widely adopted in various
sensitive applications ranging from healthcare [41, 23, 19] to finance [16,
29], the demand for interpretability has became crucial and is now even a
legal requirement in some cases [27]. While many interpretability techniques
are developed for unstructured data such as images, tabular data remains the
most common modality of data in various ML applications [9] and the majority
of computational resources are still devoted to training and deploying models
trained on this form of data [32]. Although some attempts were made to extend
the impressive results of deep-learning techniques to structured data [4, 57],
tree-based models consistently show superior performance [18]. Nevertheless,
recent work on explainable ML has focused on improving neural network
interpretability rather than tree-based models [11].

Current interpretability methods for tabular ML revolve around similar themes.
These methods are either model-agnostic or specific to tree-based models and
output an importance-score for each feature in the data. These scores measure
either a feature’s effect on predicting a single example (local) or its
contribution to the model’s overall performance (global). Therefore, the most
informative visualization component of these methods will be a simple _bar
chart_ of __feature-importance__ scores. This approach, similar to the
_feature-importance_ methods (e.g. saliency maps) used for vision and text
models [46, 54, 51], is neither truly interpretable, nor actionable [43, 34].
More so, the bar chart visualization component is even less informative
compared to image and text modalities.

A recent line of explainable ML research [34, 59, 7] on image and text data is
around providing information beyond importance scores; for instance,
visualizing important concepts (objects, colors, etc) [24]. Our goal is to
make a similar effort for tabular ML by providing visualizations that will
provide interpretations beyond simple importance scores.

###  1.1 Our contribution

We propose a new interpretability method called Feature Vectors for tabular
ML. Our contribution is orthogonal and complementary to other endeavors that
are focused on computing better _feature-importance_ “scores” i.e. scores that
are more true to the model. Feature Vectors outputs an easy-to-understand
visualization of the tabular dataset with two sets of information: 1)
_feature-importance_ scores, and 2) Feature semantics. In our definition of
feature semantics, two features are similar if they have similar interactions
with the rest of the features in predicting the outcome and therefore, are
nearly exchangeable from the model’s perspective.

Figure 1: Comparison between standard feature importance scores and Feature
Vector  The bar chart on the left shows the Gini importance score and the left
plot shows the output of Feature Vectors. The dataset is synthetic where the
features have normal distributions independent of each other. The label is
generated by applying thresholds over six of the twenty features where each
feature is a synonym to one other feature in that they are exchangeable in the
label generation rules. The magnitude of the feature vectors shows their
_feature-importance_ while the direction encodes semantics. The circles’
colors also encodes angle information to make it easier to observe the
similarity and dissimilarity of features.

###  1.2 Related Literature

Deep learning models achieve state of the art performance in tasks related to
vision, text, and speech data. Tabular data lacks spacial or temporal
invariance and therefore, despite recent effort to design deep learning models
specific to tabular data (e.g. TabNet [4] and Deep Neural Decision Trees
[57]), tree-based models still have the strongest performance [18]. Examples
of popular tree-based models are Random Forests [8], XGBoost [13], Extremely
Randomized Trees [22], Light GBM [33], and CatBoost [44].

Global interpretability methods explain the model as a whole and report the
importance of features for the model’s overall performance. In tabular ML, a
traditional approach is to compute the gain of each individual feature [37,
20, 48] by computing the amount of boost in performance ("e.g. drop in
training loss") gained by including a feature. Model-agnostic permutation-
based methods approximate a feature’s importance by randomly permuting a
feature’s value among test data points and tracking the decrease in the
model’s performance [8, 5, 52].

A large group of existing ML interpretability methods are developed for the
local scenario where the goal is to explain the model’s prediction on a single
example [39, 45, 47, 12]. The recent focus has been on gradient-based methods
[2, 50, 49] where taking the gradient of the model output with respect to the
input is possible e.g. deep neural networks. Although these methods are local,
by aggregating the importance of features across a large number of samples, it
is possible to compute global feature importance scores. An interesting recent
example is the extension of the prevalent SHAP method [39] to tree-based
models [38].

Efforts have been made to bring interpretability beyond absolute feature
importance for tabular ML. RuleFit [21] is a well-known algorithm which
translates a tree-based model into a set of binary features and then selects
the most important decision paths by training a linear classifier with
$\ell_{1}$ regularization on top of all the binary features. A different
approach is focused on computing importance scores for interaction of features
rather than individual features. [55, 53, 31, 21, 30, 28]. There have been
efforts to use visualization as a better interpreatiblity tool. Partial
Dependence Plots (PDP), Accumulate Local Effect plots (ALE) [3], Individual
Conditional Expectation (ICE) plots and centered ICE [26], visualize the
marginal effect of one or a pair of features on the output [20, 58]. Another
approach is depicting the local importance scores for a feature in a large
number of individual examples in a single plot [38]. To the best of our
knowledge, this is the first work that introduces a global interpretability
method for tabular data that visualizes semantic explanations in addition to
_feature-importance_ scores.

Figure 2: Feature Vectors A simple description of the notion behind the
Feature Vectors algorithm. The goal is to extract a meaningful embedding for
each feature by looking at its co-occurrence with other features across
sentences. The sentences are decision paths in a decision tree. In order to
impose exchangeability among features, an ensemble of decision trees are
trained such that at each split only a random subset of features is observed.
After training, all sentences are extracted and the co-occurrence of each
feature with other features in the sentences is computed using a sliding
window. THis co-occurrence matrix is then used for extracting the embeddings.
Figure 3: Feature Vector examples on three datasets. Each dot corresponds to
one feature. Features with similar angles are more interchangeable for the ML
prediction model. For example, for predicting lung cancer in the UK Biobank,
several age related features all have similar angles.

##  2 Materials and Methods

#### Semantic embedding

Our objective is to compute an embedding for each feature that contains both
importance and semantic information. Inspired by word embeddings in Natural
Language Processing (NLP), we are looking for a continuous dense
representation of features that contains semantic information. We define two
features to be semantically similar if they are exchangeable with respect to
the feature interactions that determine the model’s predictions. We want this
similarity to be reflected in the cosine similarity of their embeddings.
Similar to the common objective in NLP, enforcing exchangeability in the
context of other features will make it possible to find such a representation.
In NLP, one simple embedding technique common in NLP tasks is using the co-
occurrence of words in the vocabulary [6, 14, 40]. The intuition is that if
two words are semantically similar, in a large enough corpora of sentences,
their (normalized) co-occurrence frequency with other words will be similar.
For a large dictionary size, in order to have a low-dimensional word-
embedding, Singular Value Decomposition (SVD) dimensionality reduction is
applied to the co-occurrence matrix.

There are some problems and limitations with directly applying the idea of
word-vectors to features. First of all, unlike NLP where computing word
embeddings is possible due to virtually unlimited supply of grammatically
structured text data, in tabular data, co-occurrence of features in is not
clearly defined and there is no corpus of structured co-occurrence of
features. Luckily, in addition to their strong predictive performance, tree-
based models naturally provide us with a structured co-occurrence of features.
As shown in Fig. 2, a tree contains a number of decision paths where each path
is a conjunction of binary conditions on a number of features. Secondly,
considering a group of words with similar meanings, each word has a non-zero
chance of appearing in a sentence while in a decision tree, among a group of
similar features, only the most predictive feature will be chosen at its
respective split. For example, consider a salary prediction task where there
are two features with similar information about the outcome: _earned degree_
(categorical) and _number of years at college_ (numerical). Assuming that
earned degree is a better predictor, in a given split of the tree (e.g. in a
decision path that also contains _age_ and _marital status_ features), it will
always be chosen over _number of years at college_. As a result, it is not
possible to extract the co-occurrence of _number of years at college_ with
other features and to observe its exchangeability with _earned degree_. The
solution is to enforce a non-zero chance of occurrence to all features in a
given decision path. The solution is to train a large ensemble of trees where
each tree is only given a random subset of features (i.e. a random forest).
This is the same as using a Random Forest model.

#### Feature importance

In natural language, word embeddings are computed with a normalization
constraint so that only the semantic information is preserved in the embedding
angles [36, 56]. Our goal, however, is to preserve both importance and
semantic information. After extracting the sentences from the trained random
forest, a feature’s co-occurrence with other features is a direct indicator of
how many times it was chosen in a split. For example, between two similar
features, the less important one will only be chosen in a split if the other
feature is not present. Therefore, although the normalized co-occurrence
frequency of both features is similar, the absolute co-occurrence of the less
informative feature will be smaller. Thus, the Feature Vectors algorithm
computes word embeddings using the counts of co-occurrence.

#### Human-understandable visualization

The last desired property is to have a human-understandable visualization.
After computing the co-occurrence matrix of features, to be able to visualize
the features in an intuitive fashion, we reduce the dimensionality of word
embeddings to two. One possible drawback is that a two dimensional embedding
might lose some of the information from the feature co-occurrence pattern. In
practice, for all real-world datasets that we examined, between $80\%$ to
$99\%$ of the co-occurrence matrix’s variance is explained by the first two
components (Appendix A).

#### Feature Vectors Algorithm

The pseudo-code in Alg. 1 describes the Feature Vectors algorithm’s steps.

Algorithm 1 Feature Vectors

1: Input: A dataset $X$ with $n$ data points and $d$ features (categorical or
numerical) and the $n$ outputs $\mathbf{y}$ (categorical for classification
and numerical for regression)

2: Hyperparameters: Number of rules $R$, co-occurrence window size $w$

3: Output: Feature embeddings
$\mathbf{v}_{1},\dots,\mathbf{v}_{d}\in\mathbb{R}^{2}$

4: Initializations: Set of sentences $S=\\{\\}$

5: while $|S|<R$ do

6: Train a decision tree on $(X,\mathbf{y})$ where each split only observes a
random subset of $\lceil\sqrt{d}\rceil$ features

7: $s_{t}=\\{\mbox{decision paths in the tree}\\}$

8: $S=S\cup s_{t}$

9: $M\in\mathbb{R}^{d\times d}$: co-occurance of features within a window size
$w$ in sentences of $S$

10: $[\mathbf{v}_{1},\dots,\mathbf{v}_{d}]^{T}=$ two dimensional truncated SVD
of $M$

#### Comparison with existing methods

Existing global interpretability methods like gini importance, SHAP [39], and
permutation importance [8] will only provide scalar values for each feature
and the result is similar to Fig. 1’s bar chart. In comparison, the output of
feature vectors, while preserving the importance information, also shows the
groups of features that are semantically similar in the eyes of the model.

Unlike Feature Vectors that computes the feature embeddings using the input-
output relationship, an alternative approach is to find the feature-importance
(size) and semantic similarity (angle) separately. We can use any existing
importance method to find the size and then, if all the features are
numerical, we can use the features’ covariance matrix (instead of feature co-
occurrence) to compute the semantic similarity of features. As tabular data is
usually a mix of categorical and numerical variables, point-biserial
correlation can be used to mitigate the issue of having mixed-type variables.
We show the possible drawback of such an approach using an example in Fig. 1.
We create a synthetic dataset where the $20$ features are sampled from
independent normal distributions. The label is assigned using a set of logical
expressions from binary thresholds applied to the first $6$ input features
e.g. $(x_{1}>2\wedge x_{2}<-3)\lor x_{1}>0$ where for each true expression,
the chance of $y=1$ increases. Only $6$ features are present in the
expressions and we divide them into three pairs. The logical expressions are
created such that each group of paired features are interchangeable, that is,
for every expression that contains one of the features, similar expressions
that contain other features of that group exist. In other words, the two
features in a group are semantically similar. Fig. 1 shows that our method
perfectly discovers the pairs of semantically similar features while any
covariance-based method will fail as the features are independent.

##  3 Experiments and Results

In this section, we first show a few examples of Feature Vectors
implementation on widely used tabular datasets to show the usefulness of
information provided by our method. The second set of experiments aims to show
that the _fature-importance_ measure provided by this method are objectively
valid. We finally use the notion of Knockoff features [10] to examine the
information provided by the angle. For all experiments, we use cross-
validation for selecting the depth of trees. We empirically observe by setting
$R=100000$ (number of rules), the computed feature embeddings become stable
across multiple runs. We also set the window size to $w=3$.

#### Easy to implement

The implementation of Feature Vectors is available as a Python package and can
be accessed from https://github.com/feature-vectors/feature-vectors. This tool
is easy to use in general, and seamlessly integrates with Scikit-Learn’s [42]
tree-based models in a few lines of code:

⬇

1from featurevec import FeatureVec

2from sklearn.ensemble import RandomForestClassifier

3X, y, feature_names = data_loader()

4predictor = RandomForestClassifier().fit(X, y)

5fv = FeatureVec(

6 mode=’classify’,

7 feature_names=feature_names,

8 tree_generator=predictor

9 )

10fv.fit(X, y)

11fv.plot()

Figure 4: Knockoffs (a) Feature Vector visualizes the features and their
corresponding knockoffs. We can see that the feature and its knockoff have
similar semantics while for non-null features, the importance (embedding size)
is different. (b) A permutation test is performed. The hypothesis is that the
angle difference between each feature and its knockoff is the same as angle
difference between two randomly selected features. For each sample of the
test, features (original and knockoffs) are paired randomly. For each pair,
the difference in angle of the two features is calculated and then the
difference is averaged across all pairs. By repeating this 10000 times, the
blue histogram is generated. The red line shows the same average angular
difference but instead of random pairing of features, each feature is paired
by its knockoff. We can see that in this case, the average difference in
angles is much smaller than that of random pairing and therefore, the
hypothesis is rejected.  Figure 5: Importance scores comparison Feature
vectors importance scores are compared with Gini, Permutation, and SHAP global
importance methods. The first row shows the SDS results where we remove
features from the most important and train a new model each time and report
its test performance. The bottom row shows the SSS results where we add the
features with the order of importance and train a new model and reports its
test performance. We can see that the Feature Vectors has similar performance
to other methods.

#### Case studies on real-world datasets

We first apply the Feature Vectors method to three real-world datasets and
discuss the explanation provided by the algorithm. Feature-vectors outputs are
shown in Fig. 3 for three tabular datasets. 1) The first dataset contains the
gene expression of around 1800 genes for 3000 peripheral blood mononuclear
cells (PBMCs) from a healthy donor [1]. The cells are clustered in eight
different groups and the task is to determine the marker genes of the largest
cluster. This is achieved by first training a binary classifier that detects
if the cell is in the largest cluster by looking at the gene expression data
as input features. Once the model is trained, we can look at the feature
importance scores to determine the marker genes (e.g. top-10 genes with the
largest importance). Using Feature Vectors, however, in addition to finding
the marker genes, we have a more detailed interpretation of how genes affect
membership (Fig. 3(a)). It can be observed that the effects of marker genes
can be clustered into four major groups with similar genes. For example,
although gene HLA-DQA1 does not have the highest _feature-importance_ score,
is unique in its interaction with other genes. 2) The UK Biobank dataset has
comprehensive phenotype and genotype data of $500,000$ individuals in the UK.
The task is to predict if a person will be diagnosed with lung cancer in the
future using $341$ phenotypic features. It can be observed that the most
important features are from three main groups: A group of features that are
related to age and how much time has passed since the person stopped smoking,
a group of features that show the overall exposure to tobacco, and a group of
features that describe the activity level. 3) The last example is the
frequently used adult income prediction dataset [17, 35]. As expected, we can
see that education as a categorical feature and education-num as a numerical
feature are semantically similar.

#### Validating Feature Vector angles using Knockoffs.

Our next experiment seeks to objectively verify the angle as a valid
similarity measure. We find the notion of “Knockoff” features the best tool
for examination. In short, for a given feature, its knockoff is a fake feature
that has perfectly similar interactions with all other features in the data.
However, it only has predictive power if the original feature is removed i.e.
it is conditionally independent from the outcome. Therefore, in the Feature
Vectors plot, if the original feature has predictive power, we expect a
feature and it’s knockoff have similar angles with different magnitudes. A
formal definition is as follows: for a $d$-dimensional random variable $X$,
its knockoff $\tilde{X}$ is a random variable that satisfies:

  * •

Conditional independence from outcome: $\tilde{X}\perp\\!\\!\\!\perp Y|X$

  * •

Exchangeability $\forall S\subset\\{1,\dots,d\\}$

$[X,\tilde{X}]_{swap(S)}\;\stackrel{{\scriptstyle
d}}{{=}}\;[X,\tilde{X}]\qquad$

where $\stackrel{{\scriptstyle d}}{{=}}$ means equality in distribution and
$[X,\tilde{X}]_{swap(S)}$ means swapping the original and knockoff features in
$S$. The exchangeability condition results in perfect semantic similarity of a
feature and its knockoff while the independence condition means that if an
original feature has predictive importance (i.e. it is not null:
$X_{j}\not\\!\perp\\!\\!\\!\perp Y|X_{-j}$), its knockoff will have a smaller
importance.

Fig. 4 shows the Feature Vectors output for three datasets where the features
and their knockoffs are present (the models is trained on the concatenation of
[$X$, $\tilde{X}]$). The difficulty with knockoffs is that generating them for
an arbitrary distribution of data is not possible. In [25] it was shown that
if the distribution of $X$ is a mixture of Gaussians, one can sample the
knockoffs. Therefore, our first dataset is a synthetic $20$-dimensional
mixture of three Guassians, where only the first three features are non-null
(predictive of the outcome). As expected, in Fig. 4(a), we observe that the
features (circles) and their knockoff (squares) have similar angles.
Additionally, we can see that for non-null features, the knockoffs have a
smaller importance while for other features the knockoff and the original
feature have similar feature vectors. For the other two real-world datasets,
the validity of the generated knockoffs depends on the validity of the
Gaussian Mixture Model (GMM) assumption. Following [25], we fit a GMM model to
the data. We can see in Fig. 4(a) that the features and their knockoffs have
similar angles. In addition to the subjective observation, we perform a
permutation test in Fig. 4(b) to measure the semantic similarity of original
and knockoff features. The null hypothesis is that the average difference in
the angle between features and their knockoffs is not different from the
average difference in a random pairing of features. As expected, the
hypothesis is rejected with p-values close to $0.01$.

#### Validating _feature-importance_ scores

We conduct quantitative experiments to verify the _feature-importance_
generated by the Feature Vectors visualization. There exist various objective
metrics for measuring the goodness of global _feature-importance_ methods and
following existing work [15, 24], we will focus on two intuitive and widely
used metrics:

  * •

Smallest Destroying Subset (SDS) – Smallest subset of features that by
removing them an accurate model cannot be trained.

  * •

Smallest Sufficient Subset (SSS) – Smallest subset of features that are
sufficient for training an accurate model.

To measure SSS (SDS), we start removing (adding) features from the most
important to the least important one by one (based on each method’s importance
scores) and each time train a new model and measure its performance. Fig. 5
shows the performance of Feature Vectors compared to three other famous global
interpretability methods on four UCI ML repository datasets [17]. Other
methods are: 1) Gini Importance score which computes the number of times each
feature has been used in a split proportional to the number of samples it
splits. 2) Permutation Importance [8] which removes features one by one by
permuting its value in different data points and computes the drop in model’s
performance. 3) TreeSHAP [38], which models the features as players in a
cooperative game and computes the contribution of each feature to the
collaboration using the notion of Shapley value. We can see that Feature
Vectors has a comparable performance to the other three and as expected,
correlates very well with the Gini importance method. More examples are
provided in Appendix B. All in all, while Feature Vectors provides feature-
importance scores that are as good as other existing interpretability methods,
it provides semantic information of the features.

##  4 Discussion and Conclusions

We introduce Feature Vectors, a global interpretability method for tabular ML.
Feature Vectors takes a tabular dataset as an input and outputs a
visualization of features where each feature is shown by its two dimensional
embedding. The embeddings explain two main aspects of the data: 1) the
magnitude of the embedding gives how the importance of the feature to the
predictive power of the model. b) The direction of the corresponding feature
shows how semantically similar or dissimilar it is to other features. Semantic
similarity is defined as an interpreation tool where two features that are
semantically similar have similar interactions with other features in
predicting the outcome . Through extensive quantitative and qualitative
experiments, we show that using Feature Vectors can help users have a better
understanding of the model while not losing any information compared to other
existing methods. This work opens a new window towards creating better
interpretability tools for tabular data and using the element of visualization
as a tool for intuitive explanations of feature interactions. One interesting
future direction is to extend this global interpretability method for local
explainability as well. Feature Vectors is intrinsically limited to the tree-
based models that randomly select subsets of features (e.g. random forests)
and an important future direction is to extend the notion of feature semantics
to other models that are used for tabular data.

## References

  * [1] 10x Genomics.  3k pbmcs from a healthy donor.  10x Genomics, 2016.
  * [2] M. Ancona, E. Ceolini, C. Öztireli, and M. Gross.  Towards better understanding of gradient-based attribution methods for deep neural networks.  In International Conference on Learning Representations, 2018.
  * [3] D. W. Apley and J. Zhu.  Visualizing the effects of predictor variables in black box supervised learning models.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 82(4):1059–1086, 2020.
  * [4] S. O. Arik and T. Pfister.  Tabnet: Attentive interpretable tabular learning.  arXiv preprint arXiv:1908.07442, 2019.
  * [5] L. Auret and C. Aldrich.  Empirical comparison of tree ensemble variable importance measures.  Chemometrics and Intelligent Laboratory Systems, 105(2):157–170, 2011.
  * [6] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin.  A neural probabilistic language model.  The journal of machine learning research, 3:1137–1155, 2003.
  * [7] D. Bouchacourt and L. Denoyer.  Educe: Explaining model decisions through unsupervised concepts extraction.  arXiv preprint arXiv:1905.11852, 2019.
  * [8] L. Breiman.  Random forests.  Machine learning, 45(1):5–32, 2001.
  * [9] J. Bughin, J. Seong, J. Manyika, M. Chui, and R. Joshi.  Notes from the ai frontier: Modeling the impact of ai on the world economy.  McKinsey Global Institute, 2018.
  * [10] E. Candes, Y. Fan, L. Janson, and J. Lv.  Panning for gold:‘model-x’knockoffs for high dimensional controlled variable selection.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(3):551–577, 2018.
  * [11] D. V. Carvalho, E. M. Pereira, and J. S. Cardoso.  Machine learning interpretability: A survey on methods and metrics.  Electronics, 8(8):832, 2019.
  * [12] J. Chen, L. Song, M. Wainwright, and M. Jordan.  Learning to explain: An information-theoretic perspective on model interpretation.  In International Conference on Machine Learning, pages 883–892. PMLR, 2018.
  * [13] T. Chen and C. Guestrin.  Xgboost: A scalable tree boosting system.  In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
  * [14] R. Collobert and J. Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In Proceedings of the 25th international conference on Machine learning, pages 160–167, 2008.
  * [15] P. Dabkowski and Y. Gal.  Real time image saliency for black box classifiers.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6970–6979, 2017.
  * [16] M. F. Dixon, I. Halperin, and P. Bilokon.  Machine Learning in Finance.  Springer, 2020.
  * [17] D. Dua and C. Graff.  UCI machine learning repository, 2017.
  * [18] N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola.  Autogluon-tabular: Robust and accurate automl for structured data.  arXiv preprint arXiv:2003.06505, 2020.
  * [19] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, and S. Thrun.  Dermatologist-level classification of skin cancer with deep neural networks.  nature, 542(7639):115–118, 2017.
  * [20] J. Friedman, T. Hastie, R. Tibshirani, et al.  The elements of statistical learning, volume 1.  Springer series in statistics New York, 2001.
  * [21] J. H. Friedman, B. E. Popescu, et al.  Predictive learning via rule ensembles.  The Annals of Applied Statistics, 2(3):916–954, 2008.
  * [22] P. Geurts, D. Ernst, and L. Wehenkel.  Extremely randomized trees.  Machine learning, 63(1):3–42, 2006.
  * [23] A. Ghorbani, D. Ouyang, A. Abid, B. He, J. H. Chen, R. A. Harrington, D. H. Liang, E. A. Ashley, and J. Y. Zou.  Deep learning interpretation of echocardiograms.  NPJ digital medicine, 3(1):1–10, 2020.
  * [24] A. Ghorbani, J. Wexler, J. Y. Zou, and B. Kim.  Towards automatic concept-based explanations.  Advances in Neural Information Processing Systems, 32:9277–9286, 2019.
  * [25] J. R. Gimenez, A. Ghorbani, and J. Zou.  Knockoffs for the mass: new feature importance statistics with false discovery guarantees.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2125–2133. PMLR, 2019.
  * [26] A. Goldstein, A. Kapelner, J. Bleich, and E. Pitkin.  Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation.  Journal of Computational and Graphical Statistics, 24(1):44–65, 2015.
  * [27] B. Goodman and S. Flaxman.  European union regulations on algorithmic decision-making and a “right to explanation”.  AI magazine, 38(3):50–57, 2017.
  * [28] B. M. Greenwell, B. C. Boehmke, and A. J. McCarthy.  A simple and effective model-based variable importance measure.  arXiv preprint arXiv:1805.04755, 2018.
  * [29] J. B. Heaton, N. G. Polson, and J. H. Witte.  Deep learning for finance: deep portfolios.  Applied Stochastic Models in Business and Industry, 33(1):3–12, 2017.
  * [30] G. Hooker.  Discovering additive structure in black box functions.  In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 575–580, 2004.
  * [31] J. D. Janizek, P. Sturmfels, and S.-I. Lee.  Explaining explanations: Axiomatic feature interactions for deep networks.  Journal of Machine Learning Research, 22(104):1–54, 2021.
  * [32] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, et al.  In-datacenter performance analysis of a tensor processing unit.  In Proceedings of the 44th annual international symposium on computer architecture, pages 1–12, 2017.
  * [33] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu.  Lightgbm: A highly efficient gradient boosting decision tree.  Advances in neural information processing systems, 30:3146–3154, 2017.
  * [34] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In International conference on machine learning, pages 2668–2677. PMLR, 2018.
  * [35] R. Kohavi.  Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.  In Kdd, volume 96, pages 202–207, 1996.
  * [36] O. Levy, Y. Goldberg, and I. Dagan.  Improving distributional similarity with lessons learned from word embeddings.  Transactions of the Association for Computational Linguistics, 3:211–225, 2015.
  * [37] W.-Y. Loh.  Classification and regression trees.  Wiley interdisciplinary reviews: data mining and knowledge discovery, 1(1):14–23, 2011.
  * [38] S. M. Lundberg, G. G. Erion, and S.-I. Lee.  Consistent individualized feature attribution for tree ensembles.  arXiv preprint arXiv:1802.03888, 2018.
  * [39] S. M. Lundberg and S.-I. Lee.  A unified approach to interpreting model predictions.  In Proceedings of the 31st international conference on neural information processing systems, pages 4768–4777, 2017.
  * [40] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119, 2013.
  * [41] D. Ouyang, B. He, A. Ghorbani, N. Yuan, J. Ebinger, C. P. Langlotz, P. A. Heidenreich, R. A. Harrington, D. H. Liang, E. A. Ashley, et al.  Video-based ai for beat-to-beat assessment of cardiac function.  Nature, 580(7802):252–256, 2020.
  * [42] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Édouard Duchesnay.  Scikit-learn: Machine learning in python.  Journal of Machine Learning Research, 12(85):2825–2830, 2011.
  * [43] F. Poursabzi-Sangdeh, D. G. Goldstein, J. M. Hofman, J. W. Wortman Vaughan, and H. Wallach.  Manipulating and measuring model interpretability.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–52, 2021.
  * [44] L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin.  Catboost: unbiased boosting with categorical features.  Advances in Neural Information Processing Systems, 31, 2018.
  * [45] M. T. Ribeiro, S. Singh, and C. Guestrin.  " why should i trust you?" explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [46] M. T. Ribeiro, S. Singh, and C. Guestrin.  Model-agnostic interpretability of machine learning.  arXiv preprint arXiv:1606.05386, 2016.
  * [47] M. T. Ribeiro, S. Singh, and C. Guestrin.  Anchors: High-precision model-agnostic explanations.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [48] M. Sandri and P. Zuccolotto.  A bias correction algorithm for the gini variable importance measure in classification trees.  Journal of Computational and Graphical Statistics, 17(3):611–628, 2008.
  * [49] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [50] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [51] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [52] C. Strobl, A.-L. Boulesteix, T. Kneib, T. Augustin, and A. Zeileis.  Conditional variable importance for random forests.  BMC bioinformatics, 9(1):1–11, 2008.
  * [53] M. Sundararajan, K. Dhamdhere, and A. Agarwal.  The shapley taylor interaction index.  In International Conference on Machine Learning, pages 9259–9268. PMLR, 2020.
  * [54] M. Sundararajan, A. Taly, and Q. Yan.  Axiomatic attribution for deep networks.  In International Conference on Machine Learning, pages 3319–3328. PMLR, 2017.
  * [55] M. Tsang, S. Rambhatla, and Y. Liu.  How does this interaction affect me? interpretable attribution for feature interactions.  arXiv preprint arXiv:2006.10965, 2020.
  * [56] B. J. Wilson and A. M. Schakel.  Controlled experiments for word embeddings.  arXiv preprint arXiv:1510.02675, 2015.
  * [57] Y. Yang, I. G. Morillo, and T. M. Hospedales.  Deep neural decision trees.  arXiv preprint arXiv:1806.06988, 2018.
  * [58] Q. Zhao and T. Hastie.  Causal interpretations of black-box models.  Journal of Business & Economic Statistics, 39(1):272–281, 2021\.
  * [59] B. Zhou, Y. Sun, D. Bau, and A. Torralba.  Interpretable basis decomposition for visual explanation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 119–134, 2018.

##  Appendix A Two dimensional embeddings are sufficient

Table 1 shows the percentage of explained variance of the co-occurrence matrix
using the 2-dimensional Feature Vectors embedding.

Table 1: Explained Variance Ratio
Dataset |  Explained Variance ($\%$)
---|---
UK Biobank Lung Cancer Prediction | $81$
PBMC3k | $85$
Adult Income | $92$
Spambase | $86$
Bank Marketing | $98$
Parkinson Telemonitoring | $99$
Wine Quality | $98$
Breast Cancer | $87$

##  Appendix B Comparing _feature-importance_ methods

In Fig. 6, we compare the feature importance scores of different methods using
the SSS and SDS metrics for four more datasets.

Figure 6: Importance scores comparison Feature vectors importance scores are
compared with Gini, Permutation, and SHAP global importance methods. The first
row shows the SDS results were we remove features from the most important and
train a new model each time and report its test performance. The bottom row
shows the SSS results were we add the features with the order of importance
and train a new model and reports its test performance. We can see that the
Feature Vectors has similar performance to other methods.