# Faithful and Plausible Explanations of Medical Code Predictions

Zach Wood-Doughty, Isabel Cachola11footnotemark: 1, Mark Dredze
Johns Hopkins University
Baltimore, MD, 21211
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Equal contribution

###### Abstract

Machine learning models that offer excellent predictive performance often lack
the interpretability necessary to support integrated human machine decision-
making. In clinical medicine and other high-risk settings, domain experts may
be unwilling to trust model predictions without explanations. Work in
explainable AI must balance competing objectives along two different axes: 1)
Explanations must balance _faithfulness_ to the model’s decision-making with
their _plausibility_ to a domain expert. 2) Domain experts desire _local_
explanations of individual predictions and _global_ explanations of behavior
in aggregate.

We propose to train a proxy model that mimics the behavior of the trained
model and provides fine-grained control over these trade-offs. We evaluate our
approach on the task of assigning ICD codes to clinical notes to demonstrate
that explanations from the proxy model are faithful and replicate the trained
model behavior.

##  1 Introduction

Machine learning (ML) methods have demonstrated predictive success in medical
settings, leading to discussions of how ML systems can augment clinical
decision-making Caruana et al. (2015). However, a prerequisite to clinical
integration is the ability for healthcare professionals to understand the
justifications for model decisions. Clinicians often disagree on the proper
course of care, and share their justifications as a means of agreeing on a
treatment plan. Explainable Artificial Intelligence can enable models to
provide the explanations needed for them to be integrated into this process. A
significant challenge to explanations comes from the complexity of modern AI
models, which rely on complex deep neural networks with millions or billions
of parameters.

Similar concerns over model explanations across domains have inspired a whole
field of interpretable ML. Work in this area has considered two goals:
faithfulness – explanations that accurately convey the decision making
justifications of the model – and plausibility – explanations that make sense
to domain experts. Balancing these goals can be challenging, as more faithful
explanations that accurately convey the reasoning of complex AI systems may be
implausible to a domain expert, and vice versa. An additional trade off must
consider using sophisticated versus interpretable methods. The sophisticated
methods may yield the best performance on a task, but be least able to provide
explanations.

We propose to use a proxy model to disentangle the conflicting goals of
sophisticated methods with high accuracy and interpretability. We train a
fundamentally interpretable model – such as logistic regression – on the
_predictions_ of the target ML model, so that the behavior of the proxy model
mimics the target model’s behavior, rather than independently modeling the
target task. We then rely on the interpretable proxy model to create
explanations, allowing the trained model to use sophisticated methods to
achieve high accuracy. Validation of this approach requires answering two
questions: 1) Is the proxy faithful to the workings of the trained model? and
2) Are the produced explanations of high quality to domain experts?

We demonstrate our approach on the task of medical code prediction, following
Mullenbach et al. (2018). While ML methods have achieved predictive success on
various versions of ICD clinical code prediction, the best-performing methods
have been neural networks that are notoriously difficult to interpret.
Mullenbach et al. (2018) introduced DR-CAML, a method designed to produce
explainable predictions and they evaluated predictions with annotations from
clinicians. Their work showed that DR-CAML produces more plausible
explanations than reasonable baselines.

We reproduce their work and compare it to our approach of a proxy model.We use
a linear logistic regression proxy model that learns to mimic the behavior of
the trained DR-CAML model. We show that the proxy model is faithful to the
original model and produces plausible explanations, as measured on the
clinical annotations of generated explanations. We release our code and work
on reproducing Mullenbach et al. (2018).

##  2 Background

###  2.1 Interpretable ML

Interpretable machine learning is a major area within the growing field of
Explainable AI Doshi-Velez (2017). We present an overview of major themes in
the literature, and direct the reader to recent surveys for more details
Doshi-Velez (2017); Guidotti et al. (2018); Gilpin et al. (2018).

Past work distinguishes between “transparent” or “inherently interpretable”
models that offer their own explanations, and “post-hoc” methods that produce
explanations for a separately-trained model. Methods such as logistic
regression are often considered transparent or inherently interpretable,
because their simplicity allows a domain expert to understand how a change in
input would produce a different output Guidotti et al. (2018). However, even
simple models can prove difficult to interpret in certain settings, such as
when the model’s features are complex Lipton (2018). LIME is an example of a
post-hoc method Ribeiro et al. (2016); given a trained model of arbitrary
complexity it produces explanations for individual predictions. The trade-off
inherent and post-hoc methods is that inherently-interpretable methods are
often limited in model complexity. Deep neural networks, for example, often
demonstrate better performance but are not inherently interpretable Feng et
al. (2018), and typically rely upon post-hoc methods to derive explanations
Guidotti et al. (2018).

Lipton (2018) critiques the idea of “inherent” interpretability and argues
that methods that are intended to be transparently understood should pursue
several traits. These include simulatability, or whether a human can
reasonably work through each step of the model’s calculations to understand
how a prediction is made; decomposibility, or whether each parameter of the
model can be intuitively understood on its own; and algorithmic transparency,
or whether the model belongs to a class with known theoretical behaviors. Lou
et al. (2012) highlights linear and additive models as particularly
decomposible (or intelligible) classes of models, because “users can
understand the contribution of individual features in the model.” Our proposed
approach will use a linear model trained on bag-of-word features to provide a
simulatible, decomposible, and transparent method.

Interpretability methods are also distinguished by the form and quality of the
explanations it produces. Two primary desiderata for explanations of ML
systems are “faithfulness” and “plausibility.”111Faithfulness is also referred
to as validity or completeness; plausibility is alternatively referred to as
persuasiveness Herman (2017) See Jacovi and Goldberg (2020) for a longer
discussion of alternate terminology. A faithful method accurately describes
the true machinery of the model’s prediction, while a plausible model produces
explanations that can be interpreted by a human expert Jacovi and Goldberg
(2020). A method could be faithful but not plausible, if it accurately
explains a model’s predictions but does so in terms of high-dimensional
feature vectors that a human cannot interpret. Similarly, a method could be
plausible but not faithful if it produces concise natural language summaries
that are unrelated to the calculations that produce the model’s predictions.
Methods should attempt to achieve both goals, but there is a trade-off between
the two; explanations typically cannot be both perfectly descriptive and
concise. Plausibility, unlike faithfulness, necessarily requires an evaluation
based on human perception Herman (2017). A strength of our proposed method is
that it is designed for plausibility and transparency, but optimized for
faithfulness.

###  2.2 Explainable prediction of medical codes

Our work closely follows that of Mullenbach et al. (2018). We use the same
dataset of clinical texts and associated medical codes (described in § 4) and
compare against their method, called Description-Regularized Convolutional
Attention for Multi-Label classification (DR-CAML). DR-CAML is a neural model
that seeks to produce its own faithful explanations using a per-label
attention mechanism that highlights n-grams in the input text that were
correlated with the model’s predictions. Because DR-CAML has over six million
learned parameters, it does not fulfill simulatability or decomposability; a
single parameter cannot be understood in any intuitive way. However, the
attention mechanism allows for some insight into the model’s decision-making,
as it indicates which regions of the input text were given more weight in the
final classification.

DR-CAML’s use of attention to produce explanations has sparked discussion.
Jain and Wallace (2019) showed that attention mechanisms can provide
misleading explanations that are not faithful to the model’s true reasoning.
Wiegreffe and Pinter (2019) has argued that while the explanations produced by
attention may not always be faithful, they are often plausible. This
discussion has continued in the interpretable ML literature, with methods
demonstrating how attention mechanisms can be useful or deceptive Zhong et al.
(2019); Grimsley et al. (2020); Jain et al. (2020); Pruthi et al. (2020).
Creating models that are both faithful and plausible remains a challenge.

Figure 1: Relationship between trained DR-CAML model and proxy model. The
proxy model is trained to predict DR-CAML’s outputs, rather than the true
ICD-9 codes. This optimizes the proxy model for faithfulness.

##  3 Methods

Our proposed method is post-hoc and seeks to to balance faithfulness and
plausibility. We assume that we have a trained model with good predictive
performance, but low interpretability. Given this trained model and a dataset
on which it can be applied, we train a proxy model that takes the same input
from the dataset, but uses the trained model’s predictions as its labels. In
other words, given the dataset’s input, the proxy model predicts the outputs
of the uninterpretable model. Figure 1 gives a visual representation of the
proxy model setup. For the medical code classification task, the original
model (DR-CAML) is trained on the text of discharge summaries and produces a
probability for each of the 8,922 possible medical codes. We apply DR-CAML to
the texts in MIMIC III and save its continuous-valued probabilities as the
labels for our proxy model. By training the proxy model to output predictions
similar to that of the existing model, we are optimizing it to be faithful by
design.

We also desire that the proxy model produces plausible explanations and
fulfills the criteria from Lipton (2018): simulatibility, decomposibility, and
algorithmic transparency. To do so, we restrict our proxy model to a class of
models that fulfills these desiderata. The fundamental trade-off here is that
if we restrict our model class too much, the proxy will be unfaithful, unable
to mimic the behavior of the trained model. But if we allow for a proxy model
that is too complex, it may not provide plausible or otherwise desirable
explanations. The choice of proxy model requires some consideration of the
particular domain, as feature preprocessing and similar details may affect
what explanations are possible.

For the specific task of medical code prediction, we will use a linear
regression model trained on a bag-of-words representation of the clinical
texts. We train 8,922 proxy models, one for each medical code in the dataset’s
labels. We implement our method using the linear SGDRegressor model from
sklearn Pedregosa et al. (2011), and apply a log transform to the model’s
probability outputs and train the proxy to minimize squared loss. We release
the code for training and evaluating our
method.222github.com/isabelcachola/mimic-proxy

Our approach is similar to LIME Ribeiro et al. (2016) in that it learns a
simple (linear) model to explain a pretrained model. However, whereas LIME
learns a linear model to post-hoc explain a single prediction, our linear
model is trained to predict and explain the entire dataset of predictions.
This has several consequences. Unlike LIME, we do not require sampling
perturbed inputs that do not exist in the training data, which can produce
contrasts which are misleading or unintuitive Mittelstadt et al. (2019). Slack
et al. (2020) has shown that LIME can be fooled into providing innocuous
explanations for models that demonstrate racist or sexist behavior by
exploiting its reliance on perturbations. It also means that our proxy model
is given a more difficult task than a LIME model – it may be that a given
proxy model is insufficiently flexible to model the complexity of the
pretrained model, in which case we can measure this failure in terms of our
faithfulness evaluation (see § 4). Because LIME trains a model linear only in
the neighborhood of a given instance, its feature importance scores are
difficult to aggregate across a dataset, making extrapolation difficult van
der Linden et al. (2019). When our proxy model is faithful to the trained
model, our approach gives us explanations that we can expect to apply to
future predictions. If the proxy model demonstrates sufficient empirical
performance, a domain expert may even prefer to use it in place of the
original trained model, an option that LIME explanation models do not support.

By applying our proxy model method to the DR-CAML model from Mullenbach et al.
(2018), we enable an evaluation of both faithfulness and plausibility. We will
evaluate whether our model is faithful by seeing how closely its outputs match
the predictions of DR-CAML. Because DR-CAML was designed to be interpretable
using its attention mechanism, we can compare its explanations against those
produced by our proxy. In the next two sections, we introduce our evaluation
for the proxy model’s faithfulness to the DR-CAML model and the plausibility
of its explanations.

|  | Regression |  | Classification
---|---|---|---|---
|  |  |  |  |  | AUC | F1
Model |  | Spearman | Pearson | Kendall |  | Macro | Micro | Macro | Micro
Logistic |  | 0.036 | -0.195 | -0.135 |  | 0.734 | 0.936 | 0.012 | 0.353
Proxy |  | 0.498 | 0.794 | 0.608 |  | 0.980 | 0.995 | 0.052 | 0.416
Table 1:  Comparison of the logistic baseline and the proxy model to the DR-CAML predictions. For the F1 evaluation, we threshold the unnormalized proxy outputs at 0.5. The logistic model was trained to predict the ICD codes; the proxy model to predict DR-CAML’s predictions. As expected, the proxy model dramatically outperforms the logistic baseline in terms of faithfulness to the DR-CAML model.  | Logistic | Proxy | DR-CAML
---|---|---|---
Macro AUC | 0.596 | 0.901 | 0.906
Micro AUC | 0.889 | 0.967 | 0.972
Macro F1 | 0.033 | 0.142 | 0.224
Micro F1 | 0.278 | 0.326 | 0.536
Prec @ 8 | 0.547 | 0.483 | 0.701
Prec @ 15 | 0.413 | 0.407 | 0.548
Table 2:  Comparison of the logistic baseline, the proxy model, and DR-CAML to
true ICD labels. Although the logistic model was trained for this specific
task and the proxy model was not, the proxy model outperforms the baseline in
terms of AUC and F1. The proxy model’s outputs are unnormalized, which
partially explains the gap between its F1 scores, which are computed with a
threshold of 0.5, and its AUC scores, which are invariant to normalization.
This lack of normalization may also explain the poor performance on precision
metrics, as the proxy predicts each code independently.

##  4 Faithfulness evaluation

The MIMIC-III dataset contains anonymized English-language ICU patient
records, including physiological measurements and clinical notes Johnson et
al. (2016). Following Mullenbach et al. (2018), we focus on discharge
summaries which describe a patient’s visit and are annotated with ICD-9 codes.
There are 8,922 different ICD-9 codes that describe procedures and diagnoses
that occurred during a patient’s stay. The manual assignment of these codes to
patient records are required by most U.S. healthcare payers Topaz et al.
(2013). We duplicate the experimental setup of Mullenbach et al. (2018) which
uses the text of the discharge summaries as input to the DR-CAML model, which
then is trained to predict all ICD-9 codes associated with that document.
After applying their pre-processing code to tokenize the text, the dataset
contains 47,724 discharge summaries which are divided into training,
validation, and test splits.

Our proxy model is the combination of 8,922 linear regression models trained
to predict DR-CAML’s log probability for each ICD-9 code. After a brief grid
search on the validation set, we chose to apply L1 regularization with
$\alpha=0.0001$ for each regression. To establish that this collection of
linear regressions is faithful to the trained DR-CAML model, we want to show
that it makes similar predictions across all ICD-9 codes on held-out data.
Recall from Figure 1 that the proxy is trained not to predict the true ICD-9
codes but to output the same label probabilities as DR-CAML. In fact, the
proxy model never sees the true ICD-9 codes. We evaluate faithfulness by
comparing the outputs of DR-CAML and the proxy model on the held-out test set.
If the two systems produced identical outputs on held-out data, we would say
that the proxy was perfectly faithful. We make this comparison in three
different ways – first using regression metrics that compare the continuous
outputs of the two models, then using classification metrics with binarized
DR-CAML predictions, and finally by using the proxy model’s outputs perform
when treated as predictions for the true ICD-9 codes. For all these
comparisons, we use a logistic regression baseline that is trained to directly
predict the ICD-9 codes without knowledge of DR-CAML’s predictions. We would
expect the logistic baseline’s predictions to still be slightly correlated
with those of DR-CAML, but not to be a faithful proxy itself.

Our first evaluation uses regression metrics that assess the correlation
between the proxy’s predictions and DR-CAML’s predicted probabilities. We use
Spearman and Pearson correlation coefficients and the non-parametric Kendall
Tau rank correlation. These metrics range from -1 to 1, where perfectly
faithful predictions would have correlations of 1. These Regression results
are on the left side of Table 1.

Our second evaluation treats DR-CAML’s predictions as binary labels based on
whether they exceed the threshold used by Mullenbach et al. (2018) to compute
F1 scores. We then evaluate the faithfulness of our proxy model by treating
its outputs as unnormalized probabilities and using classification metrics
such as F1 score. These metrics range from 0 to 1, where perfectly faithful
predictions would have 1.0 AUC and F1 scores. The proxy model is considered
faithful if it correctly predicts whether DR-CAML will make a binary
prediction. We again use the logistic regression baseline. These
Classification results are on the right side of Table 1.

Finally, we use the proxy model’s predictions to predict the ground-truth ICD
code labels and compare its predictive performance against that of DR-CAML in
Table 2. While the proxy model was not trained using these labels, we can use
its predictions as unnormalized probabilities for these codes. By comparing
against the logistic regression baseline (a linear model of equal complexity),
we can see whether our training setup allows the proxy model to learn a better
predictor.

Our results show that the proxy model is quite faithful to the DR-CAML model.
Compared to the logistic regression baseline, the proxy model is dramatically
better on all metrics in Table 1. Combining the results from Tables 1 and 2 we
can see that on AUC metrics, the proxy model is closer to the DR-CAML
predictions than DR-CAML is to the ground-truth labels. The proxy model also
outperforms the logistic regression baseline in the classification metrics
(AUC and F1), indicating that the proxy model is more faithful to the DR-CAML
predictions. In Table 2, we see a large gap between its performance on the AUC
metrics and the F1 and precision metrics. This is likely because the outputs
of the proxy model are not normalized to be valid probabilities, and AUC is
invariant to normalization while and F1 and precision metrics are not.

Rudin (2019) critiques post-hoc methods in general, arguing that “if we cannot
know for certain whether our [post-hoc] explanation is [faithful], we cannot
know whether to trust either the explanation or the original model.” Because
no post-hoc method can ever be perfectly faithful to an original model, we
believe our approach to explicitly measuring faithfulness provides a useful
approach for understanding whether the proxy is “faithful enough” for a given
application. It also allows for a prediction-specific analysis – if we wish to
use the proxy model to explain a high-stakes prediction made by DR-CAML, we
can first check to see whether the two models agree upon that specific
prediction. In applications where explainability is paramount over accuracy,
our proxy model could be used as a more interpretable replacement for a high-
performing black-box model. In such a case, the evaluation of faithfulness in
Table 1 could be ignored in favor of the comparison in Table 2. We leave for
future work the question of whether the proxy model could be tuned or its
predictions could be normalized to improve its performance on F1 or other
metrics comparing it to the ground truth labels.

##  5 Plausibility Evaluation

Explanations are considered plausible if they can be reasoned about by a human
user. Thus, evaluating plausibility is typically more difficult than
faithfulness, because it requires input from human annotators Herman (2017).
An explanation that is plausible to a domain expert may not be plausible to a
layperson. Mullenbach et al. (2018) evaluated the plausibility of CAML’s
explanations by collecting annotations from a clinician. Wiegreffe and Pinter
(2019) has argued that the attention mechanism of CAML and DR-CAML generally
provide plausible explanations, even if they at times not be faithful to the
model’s internal decision-making. For each model they considered in their
paper, they extracted an explanation in the format of a 14-token subsequence
taken from the discharge summary. The clinician read all four explanations and
the correspond ICD code and rated each explanation as either “informative” or
not. They found that CAML was rated as slightly more informative than their
logistic regression and CNN baselines.

Model | Score | Interval
---|---|---
Logistic | 35 | (31, 50)
Cosine | 38 | (33, 52)
CAML | 44 | (33, 52)
DR-CAML | 48 | (34, 52)
Proxy | 65 | (39, 59)
Table 3:  Plausibility evaluation using classifier annotations. The Score
column is out of 99; the proxy model is significantly better than DR-CAML
according to a McNemar test with $p<0.01$, and better than all other models
with $p<0.001$. The Interval column shows a 95% bootstrap interval using
annotations sampled from their output probabilities. Because the classifier is
not calibrated, these sampled intervals heavily skew towards lower scores.

The format of Mullenbach et al. (2018)’s plausibility evaluation does not
easily lend itself to replication. While the authors shared their annotations
with us, the loss of important metadata made it difficult to directly
reproduce their analysis. Additionally, as the clinical annotator considered
explanations in a comparative setting, it is not trivial to compare our proxy
model against these same annotations. Our goal was to replicate this
evaluation by using a simple classifier to predict synthetic labels as to
whether the clinical domain expert would have labeled our models’ explanations
as plausible. Using BioWordVec embeddings released by Zhang et al. (2019), the
text of the ICD-9 code description, and the 14-gram explanation produced by
each model from Mullenbach et al. (2018), we train a classifier that predicts
whether an explanation would have received an “informative” or “highly
informative” label. This annotation classifier achieves an accuracy of 67.2%
and an AUC score of .726 on held-out explanations, indicating it is a useful
but noisy stand-in for the clinician. The full training and evaluation details
of this classifier are in Appendix A.3.

To conduct our plausibility evaluation, we first use or reimplement the
baseline methods from Mullenbach et al. (2018). Each model, including the
proxy, produces a 14-token explanation from the discharge summary by first
finding the 4-gram with the largest average feature importance and then
including five tokens on either side of the 4-gram. The logistic regression
baseline is the same as considered in § 4, where feature importance is
computed simply using the coefficients of the logistic model. The proxy
model’s explanations are computed in the same manner, finding the 4-gram with
the largest average coefficient weights. The CAML and DR-CAML models extract
explanations using their attention mechanism, as implemented in the code
released by Mullenbach et al. (2018). Finally, we reimplement their cosine
similarity baseline, which simply finds the 4-gram in the discharge summary
with the highest cosine similarity to the words in the ICD-9 code description.

We extract the model’s explanations for the same333Using the 99 (of 100)
discharge summaries that could be uniquely identified. See Appendix A for
details. discharge summaries as were evaluated by Mullenbach et al. (2018).
For each explanation, we use the annotation classifier described above to
predict the probability that each explanation would have been labeled as
informative. If we set the classifier threshold such that 42% of explanations
are rated as informative (matching the proportion from the original
annotations), we get the results in the Scores column of Table 3. We see that
the proxy model significantly (according to a McNemar test) outperforms all
other models. We then highlight the uncertainty introduced by relying on our
imperfect annotation classifier. Rather than thresholding the outputs of the
annotation classifier, we instead use the probabilities it outputs and sample
a set of informative labels for each explanation. We sample 1000 such sets of
labels and report the 95% confidence interval for each model’s score in the
Interval column of Table 3. Accounting for this uncertainty dramatically
reduces the differences between the methods. However, because and 95% of all
classified plausibility probabilities are between 23.6 and 61.2, this sampling
method makes it unlikely that any bootstrap interval would reach the proxy
model’s thresholded score of 65. If, instead of resampling annotations, we
vary the threshold for a plausibility label, the gap between the proxy model
and other methods widens. If we set the threshold so only 10% of explanations
are classified as informative, the proxy model achieves a score of 34, whereas
the next best model (CAML) achieves a score of only 6. Despite the inherent
uncertainty involved in extrapolating plausible scores from a fixed set of
clinical annotations, our evaluations suggest that the proxy model produces
explanations that are at least as plausible as those of the DR-CAML model on
which it is trained.

##  6 Discussion

We have introduced a method for post-hoc explanations that is designed to be
interpretable and plausible while maintaining faithfulness to the trained
model. By constraining the proxy to a class of models that is decomposible,
simulatible, and algorithmically transparent, our optimization for
faithfulness gives us a clear way to evaluate several dimensions of
interpretability. Furthermore, our proxy model only has 50K parameters,
compared to CAML’s 6 million. A key benefit of our method is its simplicity
and wide applicability. Even for a proprietary trained model for which the
learned parameters are unknown, a proxy can be trained as long as we have a
dataset that includes the trained model’s predictions. Our approach has the
additional benefit of producing a standalone proxy model that can provide
global, rather than local, explanations. Depending on the gap in predictive
performance between the proxy and original model, a skeptic of post-hoc
methods (e.g. Rudin (2019)) might prefer to discard the original model
altogether and simply make predictions using the proxy model, for which its
explanations are faithful by design.

The present work has several limitations that are left to be addressed in
future work. Though the task of medical code prediction has important
implications and has been widely studied in interpretability research, we only
consider this single task on a single English-language dataset. We believe
this proxy model approach is generally applicable as a post-hoc
interpretability method for arbitrary models, but this must be further studied
on new datasets and different trained models. It is possible that some in
future domains, trained models might be more difficult to mimic than DR-CAML.
If so, the application may require a trade-off between a less restrictive
proxy model class or a lower faithfulness of the proxy model.

Our evaluation is also limited in that it only considers a single form of
explanations: n-grams extracted via feature importances or attention weights.
Recent work has explored alternate formulations for a quality explanation
Barocas et al. (2020); some formulations may be more or less accommodating of
our proxy model method. Our plausibility evaluations rely heavily on a single
set of expert annotations from which we extrapolate using a classifier. To
demonstrate that our method can reliably provide both plausible and faithful
explanations, additional evaluations must collect new plausibility annotations
or build off of existing resources DeYoung et al. (2020).

As the ML community continues to explore new directions for interpretable
methods, definitions of desiderata may continue to evolve. Such criteria will
always depend on the domain experts who turn to an ML method for decision
support. Interpretable ML methods should clearly define how they expect to
satisfy a criterion such as faithfulness or plausibility. By designing for
plausibility and transparency and optimizing for faithfulness, our proposed
method is broadly applicable. We release our code to enable future extensions
of our work.

## Acknowledgements

We thank Sarah Wiegreffe and Jacob Eisenstein for their help and plausibility
annotations.

## References

  * Barocas et al. (2020) Solon Barocas, Andrew D Selbst, and Manish Raghavan. 2020.  The hidden assumptions behind counterfactual explanations and principal reasons.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 80–89.
  * Caruana et al. (2015) Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015.  Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.  In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1721–1730.
  * DeYoung et al. (2020) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020.  Eraser: A benchmark to evaluate rationalized nlp models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4443–4458.
  * Doshi-Velez (2017) Been Doshi-Velez, Finale; Kim. 2017.  Towards a rigorous science of interpretable machine learning.  In _eprint arXiv:1702.08608_.
  * Feng et al. (2018) Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018.  Pathologies of neural models make interpretations difficult.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3719–3728.
  * Gilpin et al. (2018) Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018.  Explaining explanations: An overview of interpretability of machine learning.  In _2018 IEEE 5th International Conference on data science and advanced analytics (DSAA)_ , pages 80–89. IEEE.
  * Grimsley et al. (2020) Christopher Grimsley, Elijah Mayfield, and Julia RS Bursten. 2020.  Why attention is not explanation: Surgical intervention and causal reasoning about neural models.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 1780–1790.
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018.  A survey of methods for explaining black box models.  _ACM computing surveys (CSUR)_ , 51(5):1–42.
  * Herman (2017) Bernease Herman. 2017.  he promise and peril of human evaluation for model interpretability.  In _Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017)_.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4198–4205.
  * Jain and Wallace (2019) Sarthak Jain and Byron C Wallace. 2019.  Attention is not explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3543–3556.
  * Jain et al. (2020) Sarthak Jain, Sarah Wiegreffe, Yuval Pinter, and Byron C. Wallace. 2020.  Learning to faithfully rationalize by construction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4459–4473, Online. Association for Computational Linguistics.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9.
  * van der Linden et al. (2019) Ilse van der Linden, Hinda Haned, and Evangelos Kanoulas. 2019.  Global aggregations of local explanations for black box models.  In _SIGIR Workshop on FACTS-IR_.
  * Lipton (2018) Zachary C Lipton. 2018.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ , 16(3):31–57.
  * Lou et al. (2012) Yin Lou, Rich Caruana, and Johannes Gehrke. 2012.  Intelligible models for classification and regression.  In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 150–158.
  * Mittelstadt et al. (2019) Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019.  Explaining explanations in ai.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pages 279–288.
  * Mullenbach et al. (2018) James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018\.  Explainable prediction of medical codes from clinical text.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1101–1111.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011.  Scikit-learn: Machine learning in python.  _the Journal of machine Learning research_ , 12:2825–2830.
  * Pruthi et al. (2020) Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020.  Learning to deceive with attention-based explanations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4782–4793, Online. Association for Computational Linguistics.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  ” why should i trust you?” explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , pages 1135–1144.
  * Rudin (2019) Cynthia Rudin. 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215.
  * Slack et al. (2020) Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020\.  Fooling lime and shap: Adversarial attacks on post hoc explanation methods.  In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , pages 180–186.
  * Topaz et al. (2013) Maxim Topaz, Leah Shafran-Topaz, and Kathryn H Bowles. 2013.  Icd-9 to icd-10: evolution, revolution, and current debates in the united states.  _Perspectives in health information management/AHIMA, American Health Information Management Association_ , 10(Spring).
  * Wiegreffe et al. (2019) Sarah Wiegreffe, Edward Choi, Sherry Yan, Jimeng Sun, and Jacob Eisenstein. 2019\.  Clinical concept extraction for document-level coding.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 261–272.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Hong Kong, China. Association for Computational Linguistics.
  * Zhang et al. (2019) Yijia Zhang, Qingyu Chen, Zhihao Yang, Hongfei Lin, and Zhiyong Lu. 2019.  Biowordvec, improving biomedical word embeddings with subword information and mesh.  _Scientific data_ , 6(1):1–9.
  * Zhong et al. (2019) Ruiqi Zhong, Steven Shao, and Kathleen McKeown. 2019.  Fine-grained sentiment analysis with faithful attention.  _arXiv preprint arXiv:1908.06870_.

| AUC | F1 | P@n
---|---|---|---
| Macro | Micro | Macro | Micro | 8 | 15
Mullenbach et al. (2018) | 0.895 | 0.986 | 0.088 | 0.539 | 0.709 | 0.561
Wiegreffe et al. (2019) | 0.889 | 0.985 | 0.080 | 0.542 | 0.712 | 0.562
Ours (using released weights) | 0.892 | 0.978 | 0.090 | 0.298 | 0.636 | 0.471
Ours (retrained) | 0.628 | 0.884 | 0.001 | 0.024 | 0.042 | 0.027
Table 4:  Published predictive performance of CAML and our replicated results.
Our experiments throughout the paper use the model with the released weights,
which is closest to the published numbers (despite Micro F1).

##  Appendix A (Re-)implementation details

###  A.1 Reproducing CAML predictive performance

The trained DR-CAML model released by Mullenbach et al. (2018) produced
predictions that matched the published F1 and ROC scores. We were unable to
precisely replicate the outputs of the CAML model. Table 4 shows the scores
published by Mullenbach et al. (2018) as well as those for a CAML
reimplementation done by Wiegreffe et al. (2019). We include the scores we
observe using the model weights released on GitHub as well as the scores for a
model we retrained from scratch. We use the released model instead of the
retrained model as its performance is much closer to the published numbers.

Model | Theirs | Ours | E1 | E2 | Full
---|---|---|---|---|---
CAML | 46 | 54 | 47 | 43 | 44
DR-CAML | – | – | 45 | 44 | 48
Cosine | 48 | 48 | 41 | 40 | 38
Logistic | 41 | 43 | 47 | 49 | 35
CNN | 36 | 46 | 51 | 47 | –
Table 5:  Plausibility evaluations and comparison to Mullenbach et al. (2018).
The Theirs column shows the published numbers; Ours shows our best attempt at
matching the clinical evaluation to the trained models. While the numbers
change dramatically, the ordering only changes by two swaps. E1 and E2 show
the results with predicted plausibility labels under two evaluation settings.
Full duplicates the results from Table 3 for comparison. While we were able to
retrain the CNN model for the E1 and E2 evaluations, the implementation was
unable to run on the GPUs we had access to.

###  A.2 Reproducing plausibility scores

The clinical plausibility annotations provided to us by the authors of
Mullenbach et al. (2018) contains the text explanations and their
corresponding annotations, but was missing the crucial metadata of which
models produced which explanations. The metadata also did not indicate from
which specific discharge summary the texts were derived; while the text
explanations were uniquely identifying for all but one of the 100 examples.
For that one example, because some patients had multiple documents sometimes
containing duplicated segments of text, there were three discharge summaries
from which the explanations could have been drawn. We thus excluded this
example from our analyses. To replicate their analysis the best we could, we
retrained or reimplemented their logistic regression, vanilla CNN, and cosine
similarity methods. We then looked at the attention or feature importance
weights for each trained model and the text explanations that had been
annotated, and assigned each model the text explanation for which it provided
the highest weight. This assignment did not perfectly align with past work:
there were six cases (out of 99) where a text explanation was ”chosen” by more
models than times it appeared as an option. Ignoring that issue and then
simply aggregating the Informative and Highly Informative clinician
annotations, we obtained the plausibility scores in the Ours column of Table
5. The Theirs column shows the published numbers from Mullenbach et al.
(2018). While the numbers change substantially, the ordering is relatively
stable with only two swaps: CAML and Cosine, and Logistic and CNN. The other
columns of the table are described below.

###  A.3 Plausibility annotation classifier

To evaluate the plausibility of our proxy model’s explanations, we trained a
classifier to predict whether an explanation would have been labeled as
plausible by the clinical domain expert. We treat this as a binary
classification task by grouping the ”Informative” and ”Highly Informative”
annotations as a single ”plausible” label. Conscious of the fact that we have
only 99 examples with four text explanations each, we use two approaches with
which to train and evaluate our classifier. The first used leave-one-out cross
validation at the example level, such that the classifier was trained on 98
examples at a time and then evaluated on the remaining one. We refer to this
evaluation as ”E1” in Table 5. The second also used leave-on-out cross
validation but at the explanation level; we held out a single text
explanation, trained on all other explanations across all examples, and then
evaluated on the held-out explanation. When an explanation appeared more than
once in a single example, we made sure to remove its duplicates from the
training data for predicting that explanation. We refer to this evaluation as
”E2” in Table 5.

The trained model is a simple logistic regression classifier trained on a
fastText embedding of both the explanation and the target ICD-9 code
description. Using the BioWordVec embeddings released by Zhang et al. (2019),
we embed each both the explanation and code description into a 200-dimensional
vector, concatenate the two vectors, and pass it to the logistic regression.
In the E1 evaluation, the model achieves an accuracy of 60.6% and an ROC AUC
score of .640. In the E2 evaluation, that increases to an accuracy of 67.2%
and an AUC score of .726, indicating that the additional within-example
explanations substantially help the classifier.

When using these classifiers to label the explanations generated by each model
instead of the plausibility scores derived in § A.2, we get the results shown
in columns E1 and E2 of Table 5.

Finally, we retrain our final classifier on all the explanations, leaving none
held out. Rather than using our classifier to evaluate the explanations that
were actually shown to the clinician, we instead use our (re-)implementation
of the four models to extract an explanation from each of the 99 discharge
summaries. These explanations thus may or may not appear in the training data
for the classifier. For this evaluation we are not worried about the
classifier overfitting, as the classifier functions as a direct replacement
for the clinician who produced the training data. The results of this analysis
are the numbers shown in Table 3 in § 5, reproduced in Table 5 in the ”Full”
column. The Logistic model does much worse on the Full evaluation than in
either E1 or E2. This may be because the explanations selected by the trained
model were worse than those selected by the model which was used for the
original clinical evaluation. We were unable to extract explanations from the
retrained CNN model, which means we were unable to evaluate it for the Full
setting.


# Optimizing contrastive learning for cortical folding pattern detection

Aymeric Gaudin  Neurospin, Bâtiment 145, CEA Saclay, 91191 Gif-sur-Yvette,
France  Louise Guillon  Neurospin, Bâtiment 145, CEA Saclay, 91191 Gif-sur-
Yvette, France  Clara Fischer  Neurospin, Bâtiment 145, CEA Saclay, 91191 Gif-
sur-Yvette, France  Arnaud Cachia  LaPsyDé, Laboratoire A.Binet-Sorbonne, 46
rue Saint Jacques, 75005 Paris  Denis Rivière  Neurospin, Bâtiment 145, CEA
Saclay, 91191 Gif-sur-Yvette, France  Jean-François Mangin  Neurospin,
Bâtiment 145, CEA Saclay, 91191 Gif-sur-Yvette, France  Joël Chavas
Neurospin, Bâtiment 145, CEA Saclay, 91191 Gif-sur-Yvette, France

###### Abstract

The human cerebral cortex has many bumps and grooves called gyri and sulci.
Even though there is a high inter-individual consistency for the main cortical
folds, this is not the case when we examine the exact shapes and details of
the folding patterns. Because of this complexity, characterizing the cortical
folding variability and relating them to subjects’ behavioral characteristics
or pathologies is still an open scientific problem. Classical approaches
include labeling a few specific patterns, either manually or semi-
automatically, based on geometric distances, but the recent availability of
MRI image datasets of tens of thousands of subjects makes modern deep-learning
techniques particularly attractive. Here, we build a self-supervised deep-
learning model to detect folding patterns in the cingulate region. We train a
contrastive self-supervised model (SimCLR) on both Human Connectome Project
(1101 subjects) and UKBioBank (21070 subjects) datasets with topological-based
augmentations on the cortical skeletons, which are topological objects that
capture the shape of the folds. We explore several backbone architectures
(convolutional network, DenseNet, and PointNet) for the SimCLR. For evaluation
and testing, we perform a linear classification task on a database manually
labeled for the presence of the ”double-parallel” folding pattern in the
cingulate region, which is related to schizophrenia characteristics. The best
model, giving a test AUC of 0.76, is a convolutional network with 6 layers, a
10-dimensional latent space, a linear projection head, and using the branch-
clipping augmentation. This is the first time that a self-supervised deep
learning model has been applied to cortical skeletons on such a large dataset
and quantitatively evaluated. We can now envisage the next step: applying it
to other brain regions to detect other biomarkers. The GitHub repository is
publicly available on https://github.com/neurospin-
projects/2022_jchavas_cingulate_inhibitory_control.

###### keywords:

Contrastive learning, SimCLR, MRI, brain, folding pattern

##  1 Introduction

The human cerebral cortex is folded and made of gyri separated by folds. Even
though the main folding patterns are common across humans, the details of the
cortical folding present a very high inter-subject variability, just like
fingerprints. This variability is both a challenge and an opportunity:
studying it can let us find anatomical biomarkers of neurodevelopmental
pathologies or of subjects’ endophenotypes linked to genetic polymorphisms.

We can decipher this variability by looking for local shape features called
cortical folding patterns or folding patterns.

Figure 1: Examples of cortical patterns. A. The power button sign pattern in
the central region is linked to epilepsy [1]. B. The anterior cingulate cortex
(ACC) has two known patterns, a double-parallel pattern and a single-fold
pattern [2]. We use the detection of these two ACC patterns to evaluate our
models.

The literature shows few cases of cortical folding patterns that have been
identified and correlated with pathologies or functions. For example, in the
precentral region, Mellerio et al. (2015) manually identified a pattern named
the Power Button Sign (PBS) pattern because of its recognizable shape (Fig.
1A). This pattern is very rare in the general population but is present in
about 60% of patients with dysplasia near the motor area, leading to epilepsy
[1]. Also, the Anterior Cingulate Cortex (ACC) has two identified folding
patterns, the single-fold pattern and the double-parallel pattern (Fig. 1B)
[3]: the presence of different ACC patterns in both hemispheres is related to
control efficiency in preschoolers [2], and the shape of the ACC folds has
been related to hallucinations among human patients suffering from
schizophrenia [4]. To complement the manual methods, Borne et al. proposed a
supervised model based on a convolutional network to detect known patterns
[5].

These manual techniques have a drawback: they are long to implement, while the
supervised ones will not permit the detection of new folding patterns. We
propose to resort to unsupervised techniques that allow the use of the large
datasets that are now available. Recently, $\beta$-VAE [6] and SimCLR [7]
models have been trained to build a latent space for folding representation
[8, 9, 10]. However, these studies have not used a manually labeled dataset
for hyperparameter tuning and evaluation.

Here, we build a contrastive learning framework to find a representation that
permits the automatic detection of relevant folding patterns. We use as inputs
the so-called cortical skeletons—topologically-defined surfaces that follow
the middle of a sulcus (Fig 3, blue surface in the left image)—, which are
stable throughout life after early childhood and are expected to be less
sensitive to data acquisition sites. We optimize a SimCLR model and evaluate
it by detecting the double-parallel pattern in the anterior cingulate cortex
of the right hemisphere.

##  2 Methods

###  2.1 Datasets

Three datasets are used in this project: the human connectome project (HCP)
dataset, the UKBioBank dataset, and the so-called ACC dataset :

  * •

Data collection and sharing for the HCP dataset was provided by the MGH-USC
Human Connectome Project [11]. We used in this work MRIs of 1101 HCP subjects,

  * •

We preprocessed 21070 subjects of the UKBioBank, which recruited 500,000
people [12],

  * •

The ACC dataset stands for the Anterior Cingulate Cortex dataset. This dataset
contains MRIs from 341 young subjects taken from 4 separate studies [13, 14,
15, 16, 17]. Each MRI has been manually labeled for the presence or absence of
the paracingulate fold, which lies parallel above the anterior cingulate fold
[18, 3]. The presence of the paracingulate fold corresponds to what we call in
this paper the double-parallel pattern.

The HCP dataset is randomly split in two, HCP-1 and HCP-2. The ACC dataset is
also randomly split in two, ACC-1 for parameter tuning and ACC-2 for testing,
using a stratification guaranteeing equipartition of the sites (as there can
be an effect of the site on MRI data acquisition), gender, and subjects having
a paracingulate fold in the right hemisphere (Fig. 2).

Figure 2: Dataset splits used for parameter tuning and evaluation.

###  2.2 Demographic information

The 341 subjects of the ACC dataset are distributed as follows: 197 subjects
are from a longitudinal study of childhood-onset schizophrenia [13, 14], out
of which 73 are schizophrenia patients, 49 are patients’ siblings, and 75 are
healthy subjects [15]. 65 subjects (46 adolescents, 19 children) are taken
from Delalande et al., 2020 [16]. Last, 79 subjects are taken from Tissier et
al., 2018 [17]. All 341 subjects have been labeled manually for the presence
of the paracingulate fold.

Gender distribution is given in Table. 1. HCP subjects were recruited from
Missouri (United States), UKBioBank subjects in 2006-2010 from across the UK,
whereas ACC subjects were from France and the USA (Table 1).

| #subjects | Age | % of male | Origin | #sites
---|---|---|---|---|---
|  | Mean [min, max] |  |  |
UKB | 21070 | 64 [44, 82] | 47 | UK | 1
HCP | 1101 | 29 [22, 40] | 48 | USA | 1
ACC | 341 | 15 [8, 40] | 58 | France, USA | 4

Table 1: Demographic information of used datasets.

###  2.3 Preprocessing

Figure 3: Preprocessing the inputs of the deep learning algorithm. Left. The
cortical skeleton (in blue) represents the cortical folds for the right
hemisphere. Middle. The cingulate crop with branch labels (one color per
branch label): each branch represents a simple surface, a junction line, or a
bottom line of a fold. Right. From the cingulate crop, we construct random
views, using either the cutout augmentation, in which only bottom branches
(represented in red) are kept inside the cutout (top right), or the branch-
clipping augmentation, in which branches are randomly removed, and all bottom
branches are removed (bottom right) .

All subjects’ brain magnetic resonance images (MRIs), affinely normalized to a
standard brain referential (ICBMc2009), were preprocessed with the
Morphologist pipeline from the BrainVISA software111https://brainvisa.info to
get their folding graph (Fig. 3), namely a graph-based representation of the
cortical skeleton. Each branch of the graph is either a simple surface, a
junction line between simple surfaces, or a bottom line of a fold; they are
connected in the graph if their corresponding surfaces touch each other [19,
20].

We focus our study on the cingulate region of the right hemisphere (Fig. 3).
For this, we compute a mask of the cingulate region over a database where the
folds were manually labeled [21]. Our final inputs are 2-mm resolution 3D
crops of dimension 17x40x38 (Fig. 3, middle).

###  2.4 Model

The contrastive model used is SimCLR, an instance discrimination contrastive
model [7]. For each input image $x$ of a batch of size N, we generate two
views, $x_{i}$ and $x_{j}$, at each epoch, whose model outputs are $z_{i}$ and
$z_{j}$, respectively. The model trains to bring together views from the same
image and moves away views of different images, in that it minimizes
$\sum_{i=1}^{N}\ell_{i,j=pos(i)}+\sum_{j=1}^{N}\ell_{j,i=pos(j)}$, with :

$\ell_{i,j}=-\log\frac{\exp(\mathrm{sim}(z_{i},z_{j})/\tau)}{\sum_{k=1,k\neq
i}^{2N}\exp(\mathrm{sim}(z_{i},z_{k})/\tau)}$ (1)

$pos(i)$ represents the positive pair associated with i, $\tau$ is a
temperature parameter, and sim(.,.) is the cosine similarity function.

###  2.5 Topology and graph-based augmentations

We test two different sets of topology-based transformations to generate the
SimCLR views (see §2.8 for details on the parameters optimization and the
right panel of Fig 3 for illustration). The first one, called ”cutout”,
produces two different views: in one view, we remove a rolling block whose
dimensions are 55% of the image volume dimensions. In the second view, all
non-zero voxels inside a random 3D block of the same size are kept, and all
non-zero voxels outside this block are removed [8]. In all views, fold bottom
pixels are conserved to keep the information on the global longitudinal shape
of the sulcus.

The second augmentation method is called ”branch-clipping”: for both views, it
removes random branches from the folding graph until there are at least 40% of
the voxels removed. All bottom voxels are removed.

We then binarize the image for all views and apply a random rotation with
nearest-neighbor interpolation with a maximum amplitude of 6° around each
axis.

###  2.6 Backbones

We tested three different backbones. The first is a simple six-layer
convolutional network, taken from a $\beta$-VAE encoder optimized for cortical
skeleton images [8]. The second one is a DenseNet architecture [22], which
gives good results when used on brain MRI data [23]. Its number of dense
blocks is reduced to two, considering the smaller image size [8].

The third is a PointNet [24]. A PointNet is a neural network that, instead of
taking a binary 3D matrix with a fixed size as input, takes point clouds,
i.e., a list of 3D coordinates corresponding to the non-zero pixels in the
original 3D matrix. Indeed, in our volumes, only 4% of the pixels have non-
zero values, making this backbone attractive. We take the original
implementation of PointNet in this work.

###  2.7 Linear classification

A standard way to assess the quality of a SimCLR model is to train classifiers
on the embeddings it generates [7]. We are using the scikit-learn
implementation of a linear Support Vector Classifier (SVC) with probability
estimates: we take as a criterion the area under the receiver operating
characteristic curve (AUC) [25].

Figure 4: Preprocessing and training pipeline. We train the SimCLR model
either on the HCP or on the UKBioBank dataset (top) and evaluate the model on
the ACC dataset, which has been manually labeled for the double-parallel
pattern in the cingulate region (bottom).

###  2.8 Deep-learning engineering and parameter optimization

We do initial grid searches using the DenseNet architecture with a
30-dimensional latent space and a non-linear projection head. We are first
doing a gridsearch on HCP-1 with evaluation on ACC-1 with a 55% cutout
augmentation [8] over the learning rate on 2, 4$\times 10^{-4}$, the batch
size on 16, 32. We then set the learning rate to 4$\times 10^{-4}$ and the
batch size to 16. We then run a second grid search on 30,45,55% cutout size
for the cutout augmentation strategy and on 30,40,50% of removed pixels for
the branch-clipping strategy. We find 55% for the cutout size and 40% for the
percentage of removed pixels to be the best.

We then test a dropout rate of 0, 0.05, 0.1 for the DenseNet backbone with
both a 4-dimension and 30-dimension latent space, with both the cutout and the
branch-clipping augmentations. We set the dropout rate to 0.05.

Last, we test the DenseNet backbone when removing or keeping the bottom
voxels. For the cutout augmentation, keeping bottom voxels increases the
probability of getting stuck in a trivial solution where all outputs lie in
the same place, particularly for latent spaces of size 4. For the branch-
clipping augmentation, removing bottom voxels improves the AUC, probably by
increasing the randomization of the views. Thus, we kept all bottom voxels in
the cutout augmentation and removed all bottom voxels in the branch-clipping
augmentation.

We then run 5 models for each condition (see §3.1). Running a ConvNet model
takes 20 minutes on a 16GB GPU (Quadro RT 5000) with 48 CPU cores.

##  3 Experiments and results

###  3.1 Optimizing SimCLR model for the detection of the double-parallel
pattern

We optimize the SimCLR self-supervised model to detect the double-parallel
pattern in the cingulate region. This is done in three main steps: we first
perform a preprocessing of the brain MRI data, we then perform parameter
optimization on the train/validation dataset, and finally, we test and
evaluate the model on the test dataset (Fig. 4).

After preprocessing the brain MRI data, we obtain crops of the cingulate
region of the right hemisphere containing only information on folds (see
§2.3).

We then optimize parameters by training the model on HCP-1 and evaluating it
using a linear SVC on ACC-1 (see§ 2.8 for details). The evaluation criterion
is the AUC for linearly classifying the double-parallel pattern in the
cingulate region. After an initial parameter initialization, we test three
backbones—a simple six-layer convolutional network (named ConvNet), a
DenseNet, and a PointNet—, three latent spaces sizes—4, 10, 30—, two
augmentation strategies—the cut-out and the branch-clipping augmentation—, and
two projection heads—linear and non-linear—. We find the best model to be the
six-layer convolutional network with a linear projection head, a latent space
size (that is, the dimension of the latent space) of 10, and using the branch-
clipping augmentation strategy (Fig. 5).

Figure 5: Parameter optimization results. Training is done on HCP-1 (half of
the HCP dataset). Evaluation is done on ACC-1 (half of the ACC dataset). Each
small marker represents a different trained model. A. AUC score as a function
of the latent space size for the ConvNet backbone. Black lines with squares
and dashed lines with down triangles stand respectively for the branch-
clipping and the cutout augmentations. The thinner solid black line
corresponds to a Principal Component Analysis (PCA) model. B. and C. are
similar plots, respectively, for the DenseNet and the PointNet backbones.

Last, for testing, we train models with the chosen hyperparameters on HCP-2
and evaluate them on ACC-2. We find the AUC of SimCLR on the latent space to
be 0.73$\pm$0.03 (n=5 models), slightly above the AUC (0.69$\pm$0.04, n=5) of
the $\beta$-VAE model optimized in [8] and applied here to the same dataset
(Fig. 6A). Even if favorable, we note that the difference doesn’t reach
statistical significance (p=0.08).

We observe that the effect of the batch size on the AUC reaches a plateau for
a small batch size of 8 (Fig. 6B); we use a batch size of 16 in our models
after an initial optimization (see §2.8). This differs from applying SimCLR on
big 2D image datasets, in which bigger batch sizes improve the results [7].
Then, we observe that, by training on the ACC-1 dataset whose size is N=171,
the result is similar to the one obtained when training the model on HCP-2
with a similar training size (p=0.84, t-test, comparing blue circle and black
square in Fig. 6B). The result is also true for the UKBioBank dataset for
enough training data, as the UKBioBank results match the HCP ones for 1101
subjects. As data acquisition sites and age ranges differ in all three
datasets, this shows that the method is age and site-resistant. Then, when
changing the training set size N, the performances increase. For UKBioBank,
the AUC increases to 0.76 when training with 21070 subjects (see Fig. 6C).

Variability (measured as the standard deviation over 5 different SimCLR
initializations) decreases from 0.04 for n=551 to 0.01 for n=21070, showing
that increasing the number of training subjects significantly stabilizes the
representation. Last, the UMAP representation (from a model trained on
UKBioBank and fit with UMAP on UKBioBank embeddings) shows how the ACC
dataset, as well as the subjects with a double-parallel pattern, is embedded
in the UKBioBank space (Fig. 6D).

Figure 6: Testing results with the best parameters for SimCLR. Training is
done on HCP-2 and evaluation is done on ACC-2. A. AUC for SimCLR compared with
the $\beta$-VAE model. B. AUC as a function of batch size. C. AUC as a
function of the training set size. The blue circle and the upper triangle
represent models for which the training is done, respectively on ACC-1 and
UKBioBank. HCP training is done exclusively on HCP-2 when the training size is
smaller than 551. D. UMAP representations of UKBioBank embeddings (top) and of
ACC embeddings (bottom). Each point represents a UKBioBank subject (green), an
ACC subject with a double-parallel sulcus (red) or without (blue.)

##  4 Discussion and conclusions

We built the software structure and applied for the first time a self-
supervised deep-learning algorithm on cortical skeletons with a database of
more than 20,000 subjects. For the first time, we implemented augmentations
that use the graph structure of the cortical folds. We optimized the model and
quantified its significance using a manually labeled dataset.

We obtained a latent representation of the cortical folds of the cingulate
region and evaluated this representation for detecting a double parallel
cingulate pattern known to be related to schizophrenia.

The best SimCLR model found has two characteristics: a very simple backbone—a
convolutional neural network—and a topology-specific augmentation—the branch-
clipping—that differs from the augmentations used in 2D image learning and
takes into account topological information of the inputs. Its simplicity
likely means the same model will work reasonably well when applied to other
brain regions.

The obtained representation quality is resilient to a change in the training
database. This advantage likely comes from using cortical skeletons as they
are less sensitive to acquisition sites and the subject’s age [15]. However,
there is still a dataset effect. A possible solution would be to apply
debiasing techniques [26] or to modify the augmentations further to reduce the
remaining site and age effect.

Now that we have a representation that has been optimized for folding pattern
detection, the next step will be to bring external information (akin to other
modalities) directly into the model to further improve its representation
quality [23]. A possible solution may be to perform a late fusion (that is, to
concatenate the modalities at the level of the latent space), followed by an
autoencoder to leverage possible correlations [27].

Acknowledgments

This work was funded by the grants ANR-19-CE45-0022-01IFOPASUBA and
ANR-20-CHIA-0027-01FOLDDICO.

## References

  * [1] Mellerio, C., Roca, P., Chassoux, F., Danière, F., Cachia, A., Lion, S., Naggara, O., Devaux, B., Meder, J.-F., and Oppenheim, C., “The Power Button Sign: A Newly Described Central Sulcal Pattern on Surface Rendering MR Images of Type 2 Focal Cortical Dysplasia,” Radiology 274, 500–507 (Feb. 2015).
  * [2] Cachia, A., Borst, G., Vidal, J., Fischer, C., Pineau, A., Mangin, J.-F., and Houdé, O., “The Shape of the ACC Contributes to Cognitive Control Efficiency in Preschoolers,” Journal of Cognitive Neuroscience 26, 96–106 (Jan. 2014).
  * [3] Yucel, M., Stuart, G. W., Maruff, P., Velakoulis, D., Crowe, S. F., Savage, G., and Pantelis, C., “Hemispheric and Gender-related Differences in the Gross Morphology of the Anterior Cingulate/Paracingulate Cortex in Normal Volunteers: An MRI Morphometric Study,” Cerebral Cortex 11, 17–25 (Jan. 2001).
  * [4] Garrison, J. R., Fernyhough, C., McCarthy-Jones, S., Haggard, M., and Simons, J. S., “Paracingulate sulcus morphology is associated with hallucinations in the human brain,” Nature Communications 6, 8956 (Nov. 2015).
  * [5] Borne, L., Rivière, D., Cachia, A., Roca, P., Mellerio, C., Oppenheim, C., and Mangin, J.-F., “Automatic recognition of specific local cortical folding patterns,” NeuroImage 238, 118208 (Sept. 2021).
  * [6] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A., “beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR (2017).
  * [7] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G., “A Simple Framework for Contrastive Learning of Visual Representations,” in [Proceedings of the 37th International Conference on Machine Learning ], 1597–1607, PMLR (Nov. 2020).
  * [8] Chavas, J., Guillon, L., Pascucci, M., Dufumier, B., Rivière, D., and Mangin, J.-F., “Unsupervised Representation Learning of Cingulate Cortical Folding Patterns,” in [Medical Image Computing and Computer Assisted Intervention – MICCAI 2022 ], Wang, L., Dou, Q., Fletcher, P. T., Speidel, S., and Li, S., eds., Lecture Notes in Computer Science, 77–87, Springer Nature Switzerland, Cham (2022).
  * [9] Guillon, L., Cagna, B., Dufumier, B., Chavas, J., Rivière, D., and Mangin, J.-F., “Detection of Abnormal Folding Patterns with Unsupervised Deep Generative Models,” in [Machine Learning in Clinical Neuroimaging ], Abdulkadir, A., Kia, S. M., Habes, M., Kumar, V., Rondina, J. M., Tax, C., and Wolfers, T., eds., Lecture Notes in Computer Science, 63–72, Springer International Publishing, Cham (2021).
  * [10] Guillon, L., Chavas, J., Bénézit, A., Moutard, M.-L., Roca, P., Mellerio, C., Oppenheim, C., Rivière, D., and Mangin, J.-F., “Identification of Rare Cortical Folding Patterns using Unsupervised Deep Learning,” Imaging Neuroscience (Jan. 2024).
  * [11] Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E. J., Yacoub, E., and Ugurbil, K., “The WU-Minn Human Connectome Project: An overview,” NeuroImage 80, 62–79 (Oct. 2013).
  * [12] Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., Liu, B., Matthews, P., Ong, G., Pell, J., Silman, A., Young, A., Sprosen, T., Peakman, T., and Collins, R., “UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age,” PLOS Medicine 12, e1001779 (Mar. 2015).
  * [13] Chakravarty, M. M., Rapoport, J. L., Giedd, J. N., Raznahan, A., Shaw, P., Collins, D. L., Lerch, J. P., and Gogtay, N., “Striatal shape abnormalities as novel neurodevelopmental endophenotypes in schizophrenia: A longitudinal study,” Human Brain Mapping 36(4), 1458–1469 (2015).
  * [14] Rapoport, J. L. and Gogtay, N., “Childhood onset schizophrenia: support for a progressive neurodevelopmental disorder,” International Journal of Developmental Neuroscience 29, 251–258 (May 2011).
  * [15] Cachia, A., Borst, G., Tissier, C., Fisher, C., Plaze, M., Gay, O., Rivière, D., Gogtay, N., Giedd, J., Mangin, J.-F., Houdé, O., and Raznahan, A., “Longitudinal stability of the folding pattern of the anterior cingulate cortex during development,” Developmental Cognitive Neuroscience 19, 122–127 (June 2016).
  * [16] Delalande, L., Moyon, M., Tissier, C., Dorriere, V., Guillois, B., Mevell, K., Charron, S., Salvia, E., Poirel, N., Vidal, J., Lion, S., Oppenheim, C., Houdé, O., Cachia, A., and Borst, G., “Complex and subtle structural changes in prefrontal cortex induced by inhibitory control training from childhood to adolescence,” Developmental Science 23(4), e12898 (2020).
  * [17] Tissier, C., Linzarini, A., Allaire-Duquette, G., Mevel, K., Poirel, N., Dollfus, S., Etard, O., Orliac, F., Peyrin, C., Charron, S., Raznahan, A., Houdé, O., Borst, G., and Cachia, A., “Sulcal Polymorphisms of the IFC and ACC Contribute to Inhibitory Control Variability in Children and Adults,” eNeuro 5, ENEURO.0197–17.2018 (Mar. 2018).
  * [18] Ono, M., Kubik, S., and Abernathey, C. D., [Atlas of the cerebral sulci ], G. Thieme Verlag ; Thieme Medical Publishers, Stuttgart; New York (1990).  OCLC: 645306373.
  * [19] Mangin, J.-F., Frouin, V., Bloch, I., Regis, J., and Lopez-Krahe, J., “From 3D magnetic resonance images to structural representations of the cortex topography using topology preserving deformations,” Journal of Mathematical Imaging and Vision 5, 297–318 (Dec. 1995).
  * [20] Rivière, D., Mangin, J.-F., Papadopoulos-Orfanos, D., Martinez, J.-M., Frouin, V., and Régis, J., “Automatic recognition of cortical sulci of the human brain using a congregation of neural networks,” Medical Image Analysis 6, 77–92 (June 2002).
  * [21] Borne, L., Rivière, D., Mancip, M., and Mangin, J.-F., “Automatic labeling of cortical sulci using patch- or CNN-based segmentation techniques combined with bottom-up geometric constraints,” Medical Image Analysis 62, 101651 (May 2020).
  * [22] Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q., “Densely Connected Convolutional Networks,” arXiv:1608.06993 [cs] (Jan. 2018).  arXiv: 1608.06993.
  * [23] Dufumier, B., Gori, P., Victor, J., Grigis, A., Wessa, M., Brambilla, P., Favre, P., Polosan, M., McDonald, C., Piguet, C. M., and Duchesnay, E., “Contrastive Learning with Continuous Proxy Meta-Data for 3D MRI Classification,” arXiv:2106.08808 [cs, stat] (June 2021).  arXiv: 2106.08808.
  * [24] Charles, R. Q., Su, H., Kaichun, M., and Guibas, L. J., “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” in [2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ], 77–85, IEEE, Honolulu, HI (July 2017).
  * [25] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E., “Scikit-learn: Machine Learning in Python,” Journal of Machine Learning Research 12(85), 2825–2830 (2011).
  * [26] Tartaglione, E., Barbano, C. A., and Grangetto, M., “EnD: Entangling and Disentangling deep representations for bias correction,” in [2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) ], 13503–13512 (June 2021).
  * [27] D’Souza, N. S., Wang, H., Giovannini, A., Foncubierta-Rodriguez, A., Beck, K. L., Boyko, O., and Syeda-Mahmood, T., “Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction in Tuberculosis,” in [MICCAI 2022 ], 13437, 287–297 (2022).