# Counterfactual contrastive learning:
robust representations via causal image synthesis

Mélanie Roschewitz<EMAIL_ADDRESS>
Imperial College London, UK Fabio De Sousa Ribeiro f.de-sousa-
<EMAIL_ADDRESS>
Imperial College London, UK Tian Xia<EMAIL_ADDRESS>
Imperial College London, UK Galvin Khara<EMAIL_ADDRESS>
Kheiron Medical Technologies, UK Ben Glocker<EMAIL_ADDRESS>
Imperial College London, UK & Kheiron Medical Technologies, UK

###### Abstract

Contrastive pretraining is well-known to improve downstream task performance
and model generalisation, especially in limited label settings. However, it is
sensitive to the choice of augmentation pipeline. Positive pairs should
preserve semantic information while destroying domain-specific information.
Standard augmentation pipelines emulate domain-specific changes with pre-
defined photometric transformations, but what if we could simulate realistic
domain changes instead? In this work, we show how to utilise recent progress
in counterfactual image generation to this effect. We propose CF-SimCLR, a
counterfactual contrastive learning approach which leverages approximate
counterfactual inference for positive pair creation. Comprehensive evaluation
across five datasets, on chest radiography and mammography, demonstrates that
CF-SimCLR substantially improves robustness to acquisition shift with higher
downstream performance on both in- and out-of-distribution data, particularly
for domains which are under-represented during training.

All code is available at https://github.com/biomedia-mira/counterfactual-
contrastive.

##  1 Introduction

The use of self-supervised learning for training medical imaging models holds
great promise. Contrastive learning (CL) in particular has proven successful
at tackling two major issues in the medical imaging domain: (i) sensitivity to
domain shift and (ii) scarcity of high-quality annotated data, across various
tasks and modalities (Azizi et al., 2021; 2023; Ghesu et al., 2022; Zhou et
al., 2023). Despite its successes, one of the main challenges when applying CL
to the medical imaging domain concerns the data augmentation pipeline, a
critical design choice that has a major impact on the downstream model
performance (Tian et al., 2020). Notably, standard CL augmentation pipelines
were designed for natural images and may be far from optimal for medical
imaging. Herein, we hypothesise that with a better augmentation pipeline,
targeted towards the modality at hand, we may substantially improve
performance, particularly in terms of robustness to domain shifts. So, how can
we generate better augmentations? Recently, we have seen significant advances
towards high-resolution, high-fidelity counterfactual image synthesis using
deep generative models (Ribeiro et al., 2023; Fontanella et al., 2023).
Intuitively, a counterfactual image can be understood as a ‘what-if’ image.
For example, given a patient’s chest radiograph, taken with scanner $A$, one
may ask “what would the image have looked like if it had been acquired with
scanner $B$?”. In medical imaging, counterfactual inference models have so far
mostly proven useful for test-time model interpretability (Augustin et al.,
2022; Sanchez et al., 2022; Atad et al., 2022; Matsui et al., 2022; Sun et
al., 2023). While notable success in image generation and faithfulness to
counterfactual conditioning has been reported (Monteiro et al., 2023; Ribeiro
et al., 2023), leveraging counterfactual images to directly improve downstream
performance in clinically relevant tasks remains to be seen.

Figure 1: Summary of the three contrastive learning strategies compared in
this study. The “do” operation denotes the counterfactual image generation
model. $\mathcal{T}$ denotes the distribution of random augmentations applied
to the images. In CF-SimCLR we systematically pair real and domain
counterfactual images to generate cross-domain positive pairs. In SimCLR+ , we
simply add the counterfactuals to the training set but do not explicitly match
real and counterfactual data during pair creation, i.e. they are independent
training samples.

In this work, we investigate the use of counterfactual image generation in
contrastive learning of image representations. Aiming to improve the
robustness of learned representations, we introduce _counterfactual
contrastive learning_ , where we combine real images and corresponding domain
counterfactuals to construct positive pairs, thereby explicitly incorporating
the counterfactual nature of the generated images into the contrastive
learning objective. To isolate the effect of counterfactual positive pair
generation, we compare our proposed counterfactual contrastive learning
strategy to the case where counterfactual images are simply added to the
training set, but not explicitly used in the contrastive loss (see Fig. 1).
Our strategy is integrated into the popular contrastive learning method SimCLR
(Chen et al., 2020) and is compatible with many other contrastive learning
methods.

Our results demonstrate that counterfactual contrastive learning substantially
improves robustness to acquisition shift. Crucially, we find that this
improvement stems from the explicit use of domain counterfactuals for positive
pair generation. Our new pretraining strategy, called CF-SimCLR, is evaluated
on two high-resolution image modalities, chest radiography and mammography,
across five different datasets and challenging classification tasks.

##  2 Related work

###  2.1 Contrastive learning

Contrastive learning consists of creating pairs of image views that share
semantic information (positive pairs), and using a contrastive objective to
learn similar representations for these pairs whilst pushing them away from
the representations of unrelated inputs (negative pairs). A seminal work in
the CL field is SimCLR proposed by Chen Chen et al. (2020), which was later
extended in various follow-up works e.g. BYOL (Grill et al., 2020), MoCo (He
et al., 2020), SWaV (Caron et al., 2020). SimCLR remains a simple, effective
and widely used method, especially in medical imaging. Azizi et al. (2023)
showed that SimCLR pretraining improves model generalisation across multiple
sources of dataset shifts. A critical choice in CL concerns the augmentations
used for (artificially) generating the contrastive views. The design of the
augmentation pipeline can substantially affect the characteristics of the
learned representations, directly impacting the downstream performance and
generalisation capabilities (Tian et al., 2020; Scalbert et al., 2023). So
far, most standard augmentation pipelines originally designed for natural
images have been used for medical imaging applications, which may be far from
optimal for addressing the intricacies of the medical imaging domain. In
medical imaging tasks, we often observe large intra-class differences due to
differences in data acquisition which can overshadow inter-class differences
such as subtle signs of disease. If not addressed, image representations
obtained with self-supervised learning such as SimCLR may encode non-relevant
differences due to acquisition shifts, which hampers downstream performance
and robustness.

###  2.2 Counterfactual image generation

Counterfactual image generation methods can be broadly divided into two
groups. First, methods focusing on interpretability, generating
“counterfactual explanations” i.e. images that would reverse the decision of a
given classifier while staying close to the original individual at test-time
(Augustin et al., 2022; Sanchez et al., 2022; Atad et al., 2022; Matsui et
al., 2022; Sun et al., 2023). The second set of methods focuses on generative
models able to directly generate ‘what-if’ images, independently of any
downstream classifier, following principles of Pearl’s ladder of causation
(Pearl, 2009). Pawlowski et al. (2020) first introduced Deep Structural Causal
Models (DSCM) to generate principled counterfactuals of small images.
Recently, Ribeiro et al. (2023) extended the DSCM framework to high-resolution
medical images, leveraging hierarchical variational autoencoders (HVAE) to
significantly improve image quality. While there has been a surge of interest
in these models, few studies have explored their usefulness for downstream
applications e.g. data augmentation in imbalanced datasets (Xia et al., 2022;
Garrucho et al., 2023), or for improving fairness (Dash et al., 2022).
Therefore, the potential of counterfactual image generation models for
improving downstream task performance remains largely under-explored.

###  2.3 Combining contrastive learning and counterfactuals

In the vision-language literature, Zhang et al. (2020) proposed to leverage
counterfactual text-image pairs in a CL objective for vision-language
grounding tasks, where the creation of negative and positive counterfactuals
is directly defined by the downstream task. In graph learning, Yang et al.
(2023) propose to leverage graph counterfactuals to generate hard negative
examples for supervised CL. To the best of our knowledge, the usefulness of
purely image-based counterfactuals in CL settings is yet to be explored.

##  3 Counterfactual contrastive learning

We introduce _counterfactual contrastive learning_ where the goal is to teach
an image encoder to ignore domain-specific image characteristics using self-
supervised learning and counterfactual inference. In standard contrastive
learning, this is the role of photometric augmentations, aiming to teach the
network to ignore contrast, brightness and other variations. However, such
simple augmentations are unable to capture realistic variations caused by
acquisition shifts (e.g., differences across scanners). Acquisition shift is
caused by many complex factors which are difficult to model with handcrafted
augmentation pipelines. We instead propose to use a counterfactual image model
to generate realistic domain changes. Our investigation can be summarised in
three steps: (i) training of a counterfactual image generation model; (ii)
pretraining an encoder in a contrastive manner (with and without
counterfactuals); (iii) evaluating the pretrained encoder with linear probing
and finetuning on challenging downstream tasks and datasets. Fig. 1 summarises
all compared contrastive pretraining strategies in this study.

###  3.1 Counterfactual image generation

To generate image counterfactuals, we build a DSCM following Ribeiro et al.
(2023) by training an HVAE conditioned on the assumed causal parents of the
image (see Section 4.2 for details on the causal graphs). The model
architecture mostly follows the one proposed by the authors for chest X-ray
counterfactuals. To further improve the quality of the HVAE we added an
embedding layer to encode raw parent variables prior to conditioning the HVAE
and slightly modified the normalisation layers to improve training
stability111All the code is made available at https://github.com/biomedia-
mira/counterfactual-contrastive. To avoid relying on external classifiers to
improve generation quality (as they may introduce their own biases), we
omitted the optional counterfactual finetuning step proposed in Ribeiro et al.
(2023), simplifying the training pipeline of our HVAE.

###  3.2 CF-SimCLR

We use SimCLR (Chen et al., 2020) as our simple yet effective CL strategy
where positive pairs are created by applying non-deterministic transformations
to every image, creating two distinct “views” that share the same semantic
information. Every view is then passed through an image encoder (e.g. ResNet)
to produce high dimensional embeddings, which are projected to a smaller
dimension with an MLP. The contrastive loss is then applied to these projected
representations $z$. For a batch of $N$ images, given one positive pair, all
other $2(N-1)$ data points are treated as negative samples. For each positive
pair, the SimCLR loss is defined as

$\mathcal{L}_{i,j}=-\log\frac{\exp(\text{sim}(z_{i},z_{j})/\tau)}{\sum_{k=1,k\neq
i}^{2N}\exp(\text{sim}(z_{i},z_{k})/\tau)},$

where $\text{sim}(u,v)=\frac{u^{T}v}{||u||||v||}$. To compute the final loss
we sum $\mathcal{L}_{i,j}$ over all positive pairs $i,j$ in the batch.

In CF-SimCLR, we propose to use domain counterfactuals to create cross-domain
positive pairs, instead of only relying on random augmentations for pair
generation. For a given image, we first generate a domain counterfactual (if
several domains are available we sample one uniformly at random). Then, both
images, the original one and its corresponding domain counterfactual, are fed
through the random augmentation pipeline; the rest of the CL pipeline is
identical to standard SimCLR, see Fig. 1. SimCLR is widely used in the medical
imaging domain (Azizi et al., 2023), however, we stress that our
counterfactual contrastive learning framework is general and directly
applicable to many other CL strategies.

##  4 Experiments

###  4.1 Datasets

We compare three pretraining strategies across two modalities and five multi-
source publicly available datasets. On chest radiography, we use PadChest
(Bustos et al., 2020), a large dataset with scans from two different scanners
for pretraining. The availability of scanner information makes this dataset an
ideal choice for pretraining and counterfactual generation. For evaluation, we
focus on pneumonia detection. First, we evaluate on in-distribution (ID) test
data from both PadChest scanners. To evaluate how well the features transfer
to out-of-distribution (OOD) domains, we use the RSNA Pneumonia Detection
(Stein et al., 2018) and CheXpert (Irvin et al., 2019) datasets. Secondly, we
test our approach on mammography, focusing on breast density prediction
(imbalanced 4-class classification). We use EMBED (Jeong et al., 2023), a
large dataset collected in the US using six different scanners. We use data
from five scanners for pretraining (referred to as ‘EMBED’ in the following)
and keep the remaining scanner as a hold-out domain for OOD evaluation (i.e.
‘Senographe Essential’). Note that in EMBED, more than 90% of the data comes
from a single scanner, which allows us to assess model performance on under-
represented domains only available in small proportions at (pre-)training
time. For additional OOD transfer evaluation, we use VinDR-Mammo (Nguyen et
al., 2022a) (two scanners, collected in Vietnam). See Table A1 in the Appendix
for details on dataset sizes, splits and inclusion criteria.

###  4.2 Counterfactual generation

(a) EMBED

(b) PadChest

Figure 2: Examples of counterfactual images generated using our model. Note
that on PadChest, text is only imprinted on a subset of Imaging scans (not on
Phillips), and our counterfactual model respects this by removing text when
generating counterfactuals from Imaging to Phillips and vice-versa. Generated
images have a resolution of 224x224 pixels for PadChest, 224x192 for EMBED.

To train our counterfactual inference model, the causal graph describing the
data generating process must be defined. For many applications, such as
interpretability, causal graphs would usually be chosen to faithfully reflect
physiological and imaging processes. In this work, we focus on generating
domain counterfactuals i.e. we only intervene on the ‘scanner’ indicator
variable. Hence, ‘scanner’ is the only variable that is explicitly required in
the causal graph, and all other factors of variation will be captured in the
exogenous noise during counterfactual inference. Importantly, the assumed
causal graphs do not include any parents relative to the downstream task (such
as disease labels), as this would violate the _unsupervised_ assumptions of
contrastive learning. For EMBED, we only included ‘scanner’ as a parent in the
DSCM, in PadChest we included ‘sex’ and ‘scanner’ (‘sex’ being optional). We
show some qualitative examples of generated domain counterfactuals in Fig. 2.
In terms of _effectiveness_ (Monteiro et al., 2023), the generated domain
counterfactuals on PadChest are able to fool a domain classifier trained on
real data 95% of the time, whereas on EMBED the model can fool the domain
classifier 80% of the time when generating counterfactuals uniformly at random
across all devices. Note that 4 out of 5 scanners represent less than 5% of
the EMBED training set, thereby increasing the difficulty of counterfactual
generation on these underrepresented scanners. To help counter this imbalance
during training we used weighted batch sampling.

###  4.3 Contrastive pretraining and evaluation

All encoders use a ResNet-50 architecture, initialised with ImageNet weights
and we use a 512 batch size for CL pretraining. To ensure fair comparisons, we
kept all hyperparameters constant across various pretraining strategies.
Following standard evaluation practices, we compare pretrained encoders with
linear probing (frozen encoder) and full model finetuning with varying amounts
of labelled data, using a weighted cross-entropy loss. All models are
finetuned with real data only. For both tasks, we evaluate models on ID data
(same dataset as for pretraining), as well as on external datasets. Evaluation
of external datasets allows us to determine whether counterfactual image
generation is useful even when the finetuning domain is disjoint from the
pretraining domains and outside of the distribution of generated
counterfactuals.

##  5 Results

In Fig. 3 and Fig. 4, we present linear probing results for 3 pretraining
strategies: (i) standard SimCLR, (ii) SimCLR+ with a training set augmented
with counterfactuals, (iii) CF-SimCLR. The difference between (ii) and (iii)
lies in the construction of positive pairs. In SimCLR+, we simply add the
counterfactuals to the training set but do not explicitly match real and
counterfactual data during pair creation, i.e. they are independent training
samples. In CF-SimCLR we systematically pair real and domain counterfactual
images as per Fig. 1.

Figure 3: Pneumonia detection results with linear probing (frozen encoder),
reported as average ROC-AUC over 3 seeds, shaded areas denote +/- one standard
error. CF-SimCLR consistently outperforms encoders trained with standard
SimCLR and SimCLR+ where counterfactuals are added to the training set.

#### Do counterfactuals improve the quality and robustness of contrastively
learned representations?

Results on pneumonia detection (Fig. 3) show that, on all datasets, CF-SimCLR
improves downstream performance compared to standard SimCLR (orange versus
blue), across all levels of label availability. Importantly, performance gains
are substantial on external datasets, even though these images were outside of
the pretraining (and counterfactual generation) domains. On mammography,
linear probing results in Fig. 4 show that CF-SimCLR consistently outperforms
standard SimCLR across all ID scanners, in all scenarios with up to 20k
labelled images. Importantly, for limited labels (<20k), CF-SimCLR performance
gains are highest for under-represented scanners (all except Selenia
Dimensions). In cases with large amounts of labelled data (>50k), encoders
perform similarly for most scanners, except for Selenia Dimensions and
Senographe Pristina where CF-SimCLR again surpasses SimCLR. On the external
VinDR dataset, for limited labels, CF-SimCLR beats both baselines on both
scanners. Importantly, on the under-represented OOD scanner (PlanMed Nuance)
CF-SimCLR performs better by a substantial margin. Findings are similar for
model finetuning with varying amounts of data (see Appendix Figs. B1 and B2).

Figure 4: Breast density results with linear probing (frozen encoder),
reported as average one-versus-rest macro ROC-AUC over 3 seeds, shaded areas
denote +/- one standard error. CF-SimCLR overall performs best across ID and
OOD data. Figure 5: t-SNE projections of embeddings from mammography encoders.
Encoders trained with standard SimCLR and SimCLR+ exhibit domain clustering.
Our CF-SimCLR embeddings are substantially less domain-separated and the only
disjoint cluster exclusively contains breasts with implants, semantically
different. Thumbnails show one randomly sampled image from each ‘implant’
cluster.

#### Is it advantageous to incorporate image counterfactuals in the
contrastive objective or is it sufficient to simply add counterfactual data to
the training set?

We find that CF-SimCLR performed consistently better than SimCLR+ where the
model is trained on the augmented training set, across datasets and most
levels of labels. While SimCLR+ sometimes improves over standard SimCLR,
performance gains are not consistent and not as large as for models trained
with CF-SimCLR (orange vs green in Figs. 3 and 4). Differences were especially
noticeable on under-represented scanners and for transfer to OOD datasets.
This aligns with the theory: the counterfactual contrastive objective
encourages explicit alignment of domains in the learned representation. An
increase in domain alignment is in turn expected to improve robustness to
acquisition shift, i.e. improve performance on under-represented scanners and
for transfer learning with limited data. We also visualise embeddings of
16,000 randomly sampled test images for each mammography encoder in Fig. 5,
showing that models trained with SimCLR as well as SimCLR+ depict very clear
domain separation in their t-SNE plots, whereas for CF-SimCLR embeddings
appear much less domain-separated.

#### What about the computational overhead?

CF-SimCLR requires training a counterfactual inference model. We want to
highlight that the generation model used in this work is lightweight compared
to other image generation models (e.g. diffusion models). Taking EMBED as an
example, we were able to generate >1M images in less than 7 hours on a single
NVIDIA 3090 GPU, requiring 20GB of VRAM. Importantly, model training was fast
(20 epochs were sufficient to yield good results). This is low compared to the
computational cost of the contrastive learning part: to train SimCLR we
required more powerful GPUs (2x NVIDIA L40 with 46GB VRAM each) and needed to
train the model for 450 epochs.

##  6 Conclusion

We propose CF-SimCLR, a counterfactual contrastive learning method to improve
the robustness of learned representations to acquisition shift. Extensive
evaluation shows that incorporating counterfactual image generation in the
contrastive learning objective substantially improves downstream performance
of pretrained models across modalities and domains; especially for under-
represented domains. Crucially, these performance gains hold even when
transferring to domains not seen during pretraining. Future work could extend
CF-SimCLR to other types of dataset shifts, e.g. enforcing subgroup invariance
to improve model fairness.

## Acknowledgements

M.R. is funded through an Imperial College London President’s PhD Scholarship.
F.R., T.X. and B.G. received funding from the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(grant agreement No 757173, project MIRA, ERC-2017-STG). B.G. also received
support from the Royal Academy of Engineering as part of his Kheiron/RAEng
Research Chair in Safe Deployment of Medical Imaging AI. We acknowledge
support of the UKRI AI programme, and the Engineering and Physical Sciences
Research Council, for CHAI - EPSRC AI Hub for Causality in Healthcare AI with
Real Data (grant number EP/Y028856/1).

## References

  * Atad et al. (2022) Matan Atad, Vitalii Dmytrenko, Yitong Li, Xinyue Zhang, Matthias Keicher, Jan Kirschke, Bene Wiestler, Ashkan Khakzar, and Nassir Navab.  Chexplaining in style: Counterfactual explanations for chest x-rays using stylegan.  _arXiv preprint arXiv:2207.07553_ , 2022.
  * Augustin et al. (2022) Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein.  Diffusion visual counterfactual explanations.  _Advances in Neural Information Processing Systems_ , 35:364–377, 2022.
  * Azizi et al. (2021) Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, et al.  Big self-supervised models advance medical image classification.  In _Proceedings of IEEE/CVF ICCV_ , pp. 3478–3488, 2021.
  * Azizi et al. (2023) Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, et al.  Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging.  _Nature Biomedical Engineering_ , pp. 1–24, 2023.
  * Bustos et al. (2020) Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria De La Iglesia-Vaya.  Padchest: A large chest x-ray image dataset with multi-label annotated reports.  _Medical image analysis_ , 66:101797, 2020.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in neural information processing systems_ , 33:9912–9924, 2020.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Dash et al. (2022) Saloni Dash, Vineeth N Balasubramanian, and Amit Sharma.  Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals.  In _Proceedings of IEEE/CVF WCAV_ , pp. 915–924, 2022.
  * Fontanella et al. (2023) Alessandro Fontanella, Grant Mair, Joanna Wardlaw, Emanuele Trucco, and Amos Storkey.  Diffusion models for counterfactual generation and anomaly detection in brain images.  _arXiv preprint arXiv:2308.02062_ , 2023.
  * Garrucho et al. (2023) Lidia Garrucho, Kaisar Kushibar, Richard Osuala, Oliver Diaz, Alessandro Catanese, Javier Del Riego, Maciej Bobowicz, Fredrik Strand, Laura Igual, and Karim Lekadir.  High-resolution synthesis of high-density breast mammograms: Application to improved fairness in deep learning based mass detection.  _Frontiers in Oncology_ , 12:1044496, 2023.
  * Ghesu et al. (2022) Florin C Ghesu, Bogdan Georgescu, Awais Mansoor, Youngjin Yoo, et al.  Self-supervised learning from 100 million medical images.  _arXiv preprint arXiv:2201.01283_ , 2022.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of CVPR_ , pp. 9729–9738, 2020.
  * Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 590–597, 2019.
  * Jeong et al. (2023) Jiwoong J Jeong, Brianna L Vey, Ananth Bhimireddy, Thomas Kim, Thiago Santos, Ramon Correa, Raman Dutt, Marina Mosunjac, Gabriela Oprea-Ilies, Geoffrey Smith, et al.  The emory breast imaging dataset (embed): A racially diverse, granular dataset of 3.4 million screening and diagnostic mammographic images.  _Radiology: Artificial Intelligence_ , 5(1):e220047, 2023.
  * Matsui et al. (2022) Teppei Matsui, Masato Taki, Trung Quang Pham, Junichi Chikazoe, and Koji Jimura.  Counterfactual explanation of brain activity classifiers using image-to-image transfer by generative adversarial network.  _Frontiers in Neuroinformatics_ , 15:802938, 2022.
  * Monteiro et al. (2023) Miguel Monteiro, Fabio De Sousa Ribeiro, Nick Pawlowski, Daniel C Castro, and Ben Glocker.  Measuring axiomatic soundness of counterfactual image models.  _arXiv preprint arXiv:2303.01274_ , 2023.
  * Nguyen et al. (2022a) Hieu T. Nguyen, Ha Q. Nguyen, Hieu H. Pham, Khanh Lam, Linh T. Le, Minh Dao, and Van Vu.  Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography.  _medRxiv_ , 2022a.  doi: 10.1101/2022.03.07.22272009.  URL https://www.medrxiv.org/content/early/2022/03/10/2022.03.07.22272009.
  * Nguyen et al. (2022b) Hieu T. Nguyen, Ha Q. Nguyen, Hieu H. Pham, Khanh Lam, Linh T. Le, Minh Dao, and Van Vu.  Vindr-mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography.  _medRxiv_ , 2022b.  doi: 10.1101/2022.03.07.22272009.  URL https://www.medrxiv.org/content/early/2022/03/10/2022.03.07.22272009.
  * Pawlowski et al. (2020) Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker.  Deep structural causal models for tractable counterfactual inference.  _Advances in Neural Information Processing Systems_ , 33:857–869, 2020.
  * Pearl (2009) Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Ribeiro et al. (2023) Fabio De Sousa Ribeiro, Tian Xia, Miguel Monteiro, Nick Pawlowski, and Ben Glocker.  High fidelity image counterfactuals with probabilistic causal models.  In _International Conference on Machine Learning_ , 2023.
  * Sanchez et al. (2022) Pedro Sanchez, Antanas Kascenas, Xiao Liu, Alison Q O’Neil, and Sotirios A Tsaftaris.  What is healthy? generative counterfactual diffusion for lesion localization.  In _MICCAI Workshop on Deep Generative Models_ , pp. 34–44. Springer, 2022.
  * Scalbert et al. (2023) Marin Scalbert, Maria Vakalopoulou, and Florent Couzinié-Devy.  Improving domain-invariance in self-supervised learning via batch styles standardization.  _arXiv preprint arXiv:2303.06088_ , 2023.
  * Stein et al. (2018) Anouk Stein, Carol Wu, Chris Carr, George Shih, Jamie Dulkowski, Jayashree Kalpathy-Cramer, Leon Chen, Luciano Prevedello, Marc Kohli, Mark McDonald, Phil Culliton, Safwan Halabi, and Tian Xia.  Rsna pneumonia detection challenge, 2018.  URL https://kaggle.com/competitions/rsna-pneumonia-detection-challenge.
  * Sun et al. (2023) Susu Sun, Stefano Woerner, Andreas Maier, Lisa M Koch, and Christian F Baumgartner.  Inherently interpretable multi-label classification using class-specific counterfactuals.  In _Medical Imaging with Deep Learning_ , 2023.
  * Tian et al. (2020) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  _Advances in neural information processing systems_ , 33:6827–6839, 2020.
  * Xia et al. (2022) Tian Xia, Pedro Sanchez, Chen Qin, and Sotirios A Tsaftaris.  Adversarial counterfactual augmentation: application in alzheimer’s disease classification.  _Frontiers in Radiology_ , 2:1039160, 2022.
  * Yang et al. (2023) Haoran Yang, Hongxu Chen, Sixiao Zhang, Xiangguo Sun, Qian Li, Xiangyu Zhao, and Guandong Xu.  Generating counterfactual hard negative samples for graph contrastive learning.  In _Proceedings of the ACM Web Conference 2023_ , pp. 621–629, 2023\.
  * Zhang et al. (2020) Zhu Zhang, Zhou Zhao, Zhijie Lin, Xiuqiang He, et al.  Counterfactual contrastive learning for weakly-supervised vision-language grounding.  _Advances in Neural Information Processing Systems_ , 33:18123–18134, 2020.
  * Zhou et al. (2023) Yukun Zhou, Mark A Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Court, et al.  A foundation model for generalizable disease detection from retinal images.  _Nature_ , pp. 1–8, 2023.

##  Appendix A Datasets splits and inclusion criteria

Table A1: Datasets splits and inclusion criteria. Splits are created at the patient level. (∗) excluding Senographe Essential, kept as separate hold-out domain. Instructions to download all datasets and reproduce all experiments can be found in our public codebase https://github.com/biomedia-mira/counterfactual-contrastive. Dataset | Inclusion criteria | Number of images
---|---|---
Train | Validation | Test
EMBED (Jeong et al., 2023) | 2D only(∗) | 223,086 | 8,295 | 65,992
Senographe Essential (Jeong et al., 2023) | - | 10,927 | 1,251 | 3,022
VinDR Mammo (Nguyen et al., 2022b) | - | 11,191 | 2,813 | 5,996
PadChest (Bustos et al., 2020) | Adult PA only | 64,989 | 7,203 | 17,993
CheXpert (Irvin et al., 2019) | PA only | 13,811 | 2,449 | 10,838
RSNA Pneumonia (Stein et al., 2018) | PA only | 8,633 | 1,524 | 4,354

##  Appendix B Finetuning results

Figure B1: Pneumonia detection results with finetuning (unfrozen encoder).
Reported as average ROC-AUC over 3 seeds, shaded areas denote +/- standard
error. We also compare to a supervised baseline initialised with ImageNet
weights. CF-SimCLR performs best overall ID and OOD. Figure B2: Breast density
results with finetuning (unfrozen encoder), reported as average one-versus-
rest macro ROC-AUC over 3 seeds, shaded areas denote +/- standard error. We
also compare to a supervised baseline initialised with ImageNet weights. CF-
SimCLR performs best overall ID and OOD.