# Reconstruction of Patient-Specific Confounders in AI-based Radiologic Image
Interpretation using Generative Pretraining

Tianyu Han  Laura Žigutytė  Else Kroener Fresenius Center for Digital Health
(EKFZ), Technical University Dresden, Dresden, Germany  Luisa Huck  Department
of Diagnostic and Interventional Radiology, University Hospital Aachen,
Germany  Marc Sebastian Huppertz  Department of Diagnostic and Interventional
Radiology, University Hospital Aachen, Germany  Robert Siepmann  Department of
Diagnostic and Interventional Radiology, University Hospital Aachen, Germany
Yossi Gandelsman  Department of Electrical Engineering and Computer Science,
UC Berkeley, CA, USA  Christian Blüthgen  Institute for Diagnostic and
Interventional Radiology, University Hospital Zurich, University of Zurich,
Zurich, Switzerland  Center for Artificial Intelligence in Medicine and
Imaging (AIMI), Stanford University, Stanford, CA, USA  Firas Khader
Department of Diagnostic and Interventional Radiology, University Hospital
Aachen, Germany  Christiane Kuhl  Department of Diagnostic and Interventional
Radiology, University Hospital Aachen, Germany  Sven Nebelung  Department of
Diagnostic and Interventional Radiology, University Hospital Aachen, Germany
Jakob Nikolas Kather  Else Kroener Fresenius Center for Digital Health (EKFZ),
Technical University Dresden, Dresden, Germany  Department of Medicine I,
University Hospital Dresden, Dresden, Germany  Medical Oncology, National
Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg,
Germany  Shared Senior Authorship  Daniel Truhn  Department of Diagnostic and
Interventional Radiology, University Hospital Aachen, Germany  Correspondence
should be addressed to T.H<EMAIL_ADDRESS>and D.T<EMAIL_ADDRESS>Shared Senior Authorship

###### Abstract

Detecting misleading patterns in automated diagnostic assistance systems, such
as those powered by Artificial Intelligence, is critical to ensuring their
reliability, particularly in healthcare. Current techniques for evaluating
deep learning models cannot visualize confounding factors at a diagnostic
level. Here, we propose a self-conditioned diffusion model termed DiffChest
and train it on a dataset of 515,704 chest radiographs from 194,956 patients
from multiple healthcare centers in the United States and Europe. DiffChest
explains classifications on a patient-specific level and visualizes the
confounding factors that may mislead the model. We found high inter-reader
agreement when evaluating DiffChest’s capability to identify treatment-related
confounders, with Fleiss’ Kappa values of 0.8 or higher across most imaging
findings. Confounders were accurately captured with 11.1% to 100% prevalence
rates. Furthermore, our pretraining process optimized the model to capture the
most relevant information from the input radiographs. DiffChest achieved
excellent diagnostic accuracy when diagnosing 11 chest conditions, such as
pleural effusion and cardiac insufficiency, and at least sufficient diagnostic
accuracy for the remaining conditions. Our findings highlight the potential of
pretraining based on diffusion models in medical image classification,
specifically in providing insights into confounding factors and model
robustness.

## Introduction

Confounding factors in medicine are often present in input data and lead to
spurious correlations that affect model predictions [1, 2, 3, 4]. For
instance, radiological datasets that are used to train a neural network for
the detection of pneumothorax, i.e., the pathologic collection of air in the
pleural space between the inner chest wall and the outer lung parenchyma,
often contain a high number of patients that have already been treated with a
chest tube. Thus the classifier model may learn to associate the presence of a
chest tube with the pneumothorax, rather than the disease itself [5]. This is
problematic, since the medical value of such a model would be the detection of
patients suffering from pneumothorax before treatment begins, i.e., before the
chest tube is inserted. The presence of a chest tube is therefore a confounder
for the presence of pneumothorax.

Recent studies on explainable AI offer ample further evidence of the presence
of confounders in clinical image interpretation [6, 7, 8, 9]. Metal tokens in
radiographs, for example, are often used to indicate the left or right side of
the body, and are a confounder for the presence of pulmonary disease such as
COVID-19 [8, 9]. However, the discovery of these confounding factors and the
visualization of possible confounders at a patient-specific level is
challenging.

One of the most promising developments in generative vision modeling are
diffusion models [10, 11, 12, 13]. These models offer a unique approach in
which the original data is perturbed towards a tractable posterior
distribution, often modeled as a Gaussian distribution. Subsequently, a neural
network is employed to generate samples from this posterior distribution [10].
In our work, we propose a self-supervised pretraining framework that leverages
diffusion models to generate patient-specific explanations for the
classification of pulmonary pathologies. After pretraining, diffusion models
facilitate the generation of image representations that are not only rich in
semantic information but also highly useful for disease classification. We
call our model DiffChest and train it on 515,704 chest radiographs (Figure 1).
The resulting model is both capable of synthesizing high-quality chest
radiographs and exhibits a convolutional feature extractor for the
classification of pulmonary pathologies. The combination of the generative
abilities with the classification capabilities of the model allows for
generating patient-specific explanations with respect to 193 conditions and
associated imaging findings. DiffChest further enables us to identify a wide
range of confounders in the training data and to evaluate model biases and
failures. We underly our model with a theoretical foundation in which we prove
that optimizing the diffusion objective is equivalent to directly maximizing
the mutual information between the input and output of the feature extractor.
Our concept presents an approach where generative pretraining can effectively
leverage vast amounts of unlabeled or noisy labeled data, enabling the
development of efficient and interpretable model-assisted diagnosis in
clinical settings.

Figure 1: Study flow. The DiffChest model was pretrained on a collection of
497,215 chest radiographs (CXRs) from California, Massachusetts, and Spain. No
labels were used to pretrain the model. Subsequently, the model’s
classification head, consisting of a single logistic regression layer,
underwent fine-tuning using a selected subset of 18,489 radiographs from the
PadChest dataset, which had been manually annotated by clinicians. The model’s
performance was evaluated on a test cohort of 7,943 radiographs from the
PadChest dataset, ensuring that these patients were not previously seen during
both the pretraining and fine-tuning stages.

## Results

### Pulmonary Disease Detection

We collected a testing cohort of 7,943 radiographs from 7,272 patients that
were labeled by expert-clinicians, see Figure 1. We then evaluated DiffChest
on this cohort for the task of finding the correct radiological diagnosis and
compared its performance to the state-of-the-art foundation model [14],
CheXzero, which previously demonstrated high performance on the PadChest
dataset [15]. CheXzero adopts contrastive learning, utilizing a dataset of
more than 370,000 chest radiographs and radiology report pairs to learn a
unified embedding space, enabling multi-label classification. Figure 2 and
Table LABEL:tab:p_ci illustrate the results of the comparison in terms of Area
Under the Curve (AUC) and 95% Confidence Intervals (CI) for 73 imaging
findings. To ensure a fair comparison, we only included findings with more
than 30 entries in the PadChest testing cohort ($N=7,943$). DiffChest
outperforms CheXzero in 60 out of the 73 imaging findings, showcasing its
superior performance. The average AUC improvement achieved by DiffChest over
CheXzero is 0.110, with mean AUC values of 0.791 and 0.681, respectively.
DiffChest achieves an AUC of at least 0.900 for 11 imaging findings and at
least 0.700 for 59 imaging findings out of the total 73 imaging findings
evaluated, further underlining its potential applicability in clinical
scenarios.

Figure 2: AI-based detection of imaging findings by DiffChest and the CheXzero
foundation model. The mean AUC and 95% CI is shown for each imaging finding
with more than 30 entries in the PadChest testing cohort. Findings are sorted
by the mean AUC of DiffChest. Top 36 imaging findings are shown in panel a,
and the rest imaging findings are shown in panel b. Among the total 73
evaluated imaging findings, DiffChest achieves a mean AUC of greater than
0.900 for 11 findings and greater than 0.700 for 59 findings. $n$ in this plot
refers to the number of positive examinations in the PadChest test set.
DiffChest outperforms CheXzero in 60 out of the 73 imaging findings.
CVCJ$=$central venous catheter via jugular vein; CVCS$=$central venous
catheter via subclavian vein; nsg tube$=$nasogastric tube; copd$=$chronic
obstructive pulmonary disease.

### Generating Patient-Specific Explanations

AI in medicine must be explainable to be integrated into clinical practice
[16]. While approaches like feature attribution [17], class activation mapping
(CAM) [18], and Grad-CAM [19] provide insights by highlighting regions or
pixels in an image that influence the classification, they fall short of
explaining the specific attributes within those regions that are relevant to
the classification. They do not clarify whether the classification of medical
images is influenced by factors such as the radiographic density of an organ
or its shape. Here, we provide a visual explanation for DiffChest’s
classification decisions by starting from the representation of the original
image in the semantic latent space $\mathbf{z}$ and then modifying this
representation such that the image synthesized by the diffusion model has a
higher probability of being classified as exhibiting the pathology (see
Extended Data Figure 1c and Figure 3). In other words, we produce a synthetic
image that represents the same patient and that visualizes how the radiograph
of that patient would look to the neural network if it had that pathology.

In detail, in the reverse diffusion kernel
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z}^{\ast})$, we add
adversarial noise to the last logistic regression layer for the encoded
feature $\mathbf{z}$:

$\mathbf{z}^{\ast}:=\mathbf{z}-\epsilon\cdot\nabla_{\mathbf{z}}\mathcal{L}(h(\mathbf{z};\theta_{lr}),y^{t}),$
(1)

where $\mathcal{L}$ is the cross-entropy loss, $h(\mathbf{z};\theta_{lr})$ is
the logistic regression layer, and $y^{t}$ is the target label. Unlike
adversarial examples [20], which are generated by perturbing the input image,
our approach generates counterfactual examples by perturbing the latent code
$\mathbf{z}$, which is then used to generate a new image containing the
desired clinical attributes [21]. In Figure 3 and Figure S3, S4 and S5, our
explanations answer the question: "Had the original $x$ been $x^{t}$, then the
classifier prediction would have been $y^{t}$ instead of the original class
$y$," where $x^{t}$ is the target class explanation and $y^{t}$ is the target
label.

This paradigm is important for the introduction of AI models in medicine as it
allows users to understand the model’s decision-making by getting a direct
visual feedback on which image characteristics lead to the model classifying a
radiograph as exhibiting a certain pathology. Examples visualized in Figure 3
comprise overinflated lungs for patients suffering from chronic obstructive
pulmonary disease, or characteristic changes of the lung parenchyma for
patients suffering from interstitial pneumonic infiltration. These
visualizations might also allow the clinical expert to make more nuanced
diagnoses by emphasizing different pneumonic patterns (compare interstitial
pattern, alveolar pattern, and reticulonodular interstitial pattern) that
would not have been possible with traditional methods of AI explainability
such as CAM.

Figure 3: Visual explanations of patient CXRs. The original radiograph is
highlighted by a black frame, which was acquired from a male patient born in
1928. It was labeled as Normal in the PadChest dataset. To demonstrate that
DiffChest incorporates the image characteristics of a multitude of
pathologies, we synthesize nine different pathologies, namely chronic
obstructive pulmonary disease signs, hiatal hernia, interstitial pneumonia,
alveolar pneumonia, aortic elongation, emphysema, axial hyperostosis,
reticulonodular interstitial pneumonia, and costophrenic angle blunting for
this patient. The generated explanations exhibit relevant pathological
signatures, which are highlighted with arrows, stars, and frames by a board-
certified radiologist with 12 years of experience.

### Confounders in the Training Data can be Identified

Figure 4: Confounders as synthesized by DiffChest. DiffChest achieves strong
performance when acting as a classification model and detecting foreign
materials (a) and the presence of pathologies as exemplified by a high AUC
(c). However, the model may rely on shortcut signals, i.e., confounders, to
achieve this performance. We, therefore, employed DiffChest to synthesize
radiographs that exhibit a specific pathology based on a non-pathological
radiograph (b and d). The specific (target) pathology is highlighted by a
white frame, while confounders are highlighted by a red frame. Typical
confounding signals contained sternal wire cerclages, drainage tubes, device
wirings, and catheters. Importantly, DiffChest also created synthetic
radiographs that depict co-morbidities as shown in (d).

AI models are able to exploit subtle image characteristics for their
diagnosis. While this may be helpful to achieve super-human performance, it
comes with a problem: When the AI model is trained to differentiate between
two groups (e.g., patients with pneumonia and without pneumonia) it may focus
on differences in the group that are not directly related to the pathology,
but rather are present as spurious correlations. For instance, patients
suffering from pneumonia might more often have a catheter inserted, which is
not directly related to pneumonia but rather to the treatment of pneumonia.
Therefore, an AI model which is trained to detect pneumonia may in fact learn
to detect the presence of a catheter, which is a confounder [6, 7, 8, 9].
DiffChest’s visual explanations allow us to identify such confounders by
testing if these spurious correlations are present in these synthetic images.

To demonstrate this, we performed the following reader experiments: We let
DiffChest generate synthetic radiographs that exhibited a specific pathology
based on a non-pathological radiograph. Subsequently, a board-certified
radiologist with twelve years of experience checked for the presence of
confounders in these images. Confounders were classified in either being
treatment-related (such as the presence of a catheter for patients suffering
from pneumonia, Figure 4b) or as being related to the pathology (such as the
presence of a lung congestion for patients suffering from cardiomegaly, Figure
4d). Figure 4 shows exemplary results for the generated synthetic images
visualizing the target pathology and along with the confounders.

To verify if our model can accurately pick up a confounding bias in the
original data, we performed the second reader study in which we asked three
radiologists to annotate the presence of confounders in the original data, see
Extended Data Figure 2 for a detailed breakdown of the study design. The
results of this experiment are given in Tables Confounders in the Training
Data can be Identified and Confounders in the Training Data can be Identified
that give the number of real radiographs for which at least one treatment-
related confounder (Table Confounders in the Training Data can be Identified)
or at least one physiology-related confounder (Table Confounders in the
Training Data can be Identified) was present. We illustrated the discordance
between the radiologists when identifying treatment-related and physiology-
related confounders in Extended Data Figure 4 and 5. We found that the
confounders that had been identified in the synthetic images were also present
in a large proportion of the real radiographs, thereby proving that DiffChest
is able to identify confounders in the data.

Table 1: Presence of treatment-related confounders in radiographs

"Diagnosis" in Radiograph | Radiographs (n) | Confounder present | Ratio (%) | Fleiss’ Kappa
---|---|---|---|---
artificial heart valve | 30 | 30 | 100.0 | 1.000
central venous catheter via jugular vein | 30 | 29 | 96.7 | 0.900
endotracheal tube | 30 | 28 | 94.4 | 0.867
heart insufficiency | 30 | 24 | 82.2 | 0.731
artificial aortic heart valve | 24 | 22 | 91.7 | 1.000
pleural effusion | 30 | 12 | 42.2 | 0.967
lung metastasis | 30 | 6 | 21.1 | 0.967
pneumothorax | 30 | 5 | 16.7 | 1.000
cardiomegaly | 30 | 4 | 13.3 | 0.932
calcified densities | 30 | 3 | 11.1 | 0.861
catheter | 9 | 1 | 11.1 | 1.000

In total, 303 radiographs were analyzed by three radiologists for the presence
of a subset of confounders (Extended Data Figure 3). The subset of confounders
that these radiographs were tested for had previously been identified with the
help of DiffChest. Fleiss’ kappa was computed to quantify inter-reader
agreement.

Table 2: Presence of physiology-related confounders in radiographs

"Diagnosis" in Radiograph | Radiographs (n) | Confounder present | Ratio (%) | Fleiss’ Kappa
---|---|---|---|---
central venous catheter via jugular vein | 30 | 20 | 66.7 | 0.551
pleural effusion | 30 | 20 | 66.7 | 0.660
cardiomegaly | 30 | 17 | 58.9 | 0.766
calcified densities | 30 | 14 | 46.7 | 0.466
lung metastasis | 30 | 11 | 37.8 | 0.358
catheter | 9 | 2 | 22.2 | 1.000

In total, 159 radiographs were analyzed by three radiologists for the presence
of a subset of confounders (Extended Data Figure 3). The subset of confounders
that these radiographs were tested for had previously been identified with the
help of DiffChest. Fleiss’ kappa was computed to quantify inter-reader
agreement.

### Mutual Information Maximization for Free

Diffusion models belong to a class of latent variable models expressed as
$p_{\theta}(\mathbf{x}_{0})=\int
p_{\theta}\left(\mathbf{x}_{0:T}\right)d\mathbf{x}_{1:T}$. In this
formulation, $p_{\theta}(\mathbf{x}_{i})$ refers to a neural network
responsible for denoising a latent variable $\mathbf{x}_{i}$ at time $t=i$, as
shown in Extended Data Figure 1a. The sequence of latent variables
$\mathbf{x}_{1}$, $\mathbf{x}_{2}$, …, $\mathbf{x}_{T}$ shares the same
dimensionality as the observed data $\mathbf{x}_{0}$, sampled from the data
distribution $p(\mathbf{x}_{0})$ [10]. However, such a high-dimensional latent
space poses challenges in finding meaningful directions, i.e., directions
which are identified by human experts as relevant to the disease. This
property is, however, essential for representation control [22, 23]. In
DiffChest, we explicitly introduce an additional 512-dimensional latent space
$p_{\phi}(\mathbf{z}|\mathbf{x}_{0})$, which conditions the reverse diffusion
process $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z})$, to obtain a
latent space in which we can identify meaningful directions. We will refer to
this latent space as semantic latent space [24].

Following the approach described in [10] and [24], DiffChest was trained using
a simplified diffusion objective:

$L_{\text{simple}}=\sum_{t=1}^{T}\mathbb{E}_{\mathbf{x}_{0},\mathbf{\epsilon}_{t}}\left[\left\|\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t,\mathbf{z})-\mathbf{\epsilon}_{t}\right\|_{2}^{2}\right].$
(2)

In Section Extended derivations, we prove that optimizing Equation 2 is
equivalent to maximizing the likelihood of a joint distribution between
${\mathbf{x}_{0},\mathbf{z}}$, i.e., $\log
p_{\theta,\phi}(\mathbf{x}_{0},\mathbf{z})$. This joint probability can be
further decomposed into the product of the likelihood of the data
$p_{\theta}(\mathbf{x}_{0})$ and a second term
$p_{\theta,\phi}(\mathbf{z}|\mathbf{x}_{0})$, which is the posterior
distribution of $\mathbf{z}$ given $\mathbf{x}_{0}$. Here, the second term
$\log p_{\theta,\phi}(\mathbf{z}|\mathbf{x}_{0})$ is also known as a lower
bound of mutual information in Variational Information Maximization (VIM) [25,
26]:

$I(\mathbf{z};\mathbf{x}_{0})\geq\mathbb{E}_{(\mathbf{z},\mathbf{x_{0}})\sim
p(\mathbf{z},\mathbf{x_{0}})}\left[\log
p_{\theta,\phi}(\mathbf{z}|\mathbf{x}_{0})\right]+H(\mathbf{z}),$ (3)

We denote $I(\mathbf{z};\mathbf{x}_{0})$ as the mutual information between
$\mathbf{z}$ and $\mathbf{x}_{0}$, while $H(\mathbf{z})$ represents the
entropy of $\mathbf{z}$. Therefore, we can interpret the training of DiffChest
as maximizing both the data likelihood $\log p_{\theta}(\mathbf{x}_{0})$ and
the lower bound of the mutual information between the input image
$\mathbf{x}_{0}$ and its encoded latent code $\mathbf{z}$. Once trained,
DiffChest excels at generating high-quality samples while also preserving the
semantic manipulation of the input data.

## Discussion

In this work, we used a diffusion pretraining method for medical image
interpretation that allows to train classifiers without requiring large
amounts of annotated data while also giving detailed insights into the model’s
decision-making process. The model development involved large-scale
pretraining on 497,215 unlabeled chest radiographs followed by a subsequent
finetuning step on a small subset of annotated data. The results show that,
with limited finetuning, DiffChest outperforms CheXzero [15], an expert-level
foundation model, in detecting 60 out of 73 pathologic imaging findings of the
chest. Our method maintains high performance even when finetuning on only 3%
of the total finetuning data, demonstrating its data efficiency.

In addition, we demonstrated that the diffusion part of the model can be used
in conjunction with the classification part to synthesize high-quality
radiographs. This capability can serve two purposes: First, it can explain the
model’s reasoning to its users and second it can be used to identify
confounders in the training data.

This is important since medical practitioners need to have confidence in the
rationale behind an AI model’s predictions before they will consider
delegating aspects of their workload to the system. The ability to view a
synthesized image that highlights the features upon which the model focusses
can facilitate a quick assessment by physicians to determine whether the AI
model’s diagnosis aligns with their own expert judgment. Moreover, the issue
of confounding variables poses a significant challenge in the development of
AI models for medical applications, leading to potential model biases [27].
The synthesization of radiographs by our model allows the direct
identification of these confounders within the training dataset, thereby
offering an avenue to enhance data quality and to build more robust machine
learning models. We substantiated this claim through a reader study, wherein
we demonstrate that the confounders identified by DiffChest were consistently
present in a large subset of real radiographic images.

In contrast to currently prevailing contrastive learning approaches [15, 28],
which rely on the information noise contrastive estimation loss to maximize
the mutual information between the input and the context vector [29, 30, 31,
32], we theoretically prove that optimizing the simplified diffusion objective
while conditioning on the latent vector $\mathbf{z}$ is equivalent to
maximizing both the data likelihood and the mutual information between the
input and its encoded feature. This theoretical insight lends support to the
application of diffusion pretraining across the broader field of medical
imaging.

However, our model’s ability to manipulate attributes of real clinical images
raises ethical concerns, similar to DeepFake [33]. This potential may in
principle be exploited for the generation of fake medical data which may be
leveraged for insurance fraud. This is a problem since the detectability of
manipulated samples is challenging as compared to standard adversarial
approaches. A potential solution to this problem is the addition of another
neural network for detection of fake images, as suggested by Preechakul et al.
[24]. Our work has limitations in that the resolution of the synthesized
images is lower than what is normally used in clinical practice. This may be
solved by integrating progressive growing techniques as demonstrated by Ho et
al. [34] and Karras et al. [35]. However, implementing these techniques may
require substantial hardware resources, which may not always be readily
available in clinical settings. Exploring resource-efficient alternatives to
improve generative capabilities while maintaining practical feasibility will
therefore be an important future research direction.

In conclusion, this study demonstrates the potential of generative pretraining
for developing interpretable models in medical image analysis. The model’s
ability to identify confounders in the training data and analyze model biases
and failures is a significant step towards ensuring reliable and accountable
AI applications in healthcare. Beyond chest radiography, our proof-of-concept
study highlights the broader potential of self-supervised learning in
conjunction with diffusion models, offering a pathway to informed predictions
for medical personnel.

## Methods

### Ethics statement

This study was carried out in accordance with the Declaration of Helsinki and
local institutional review board approval was obtained (EK028/19).

### Datasets

Our method was trained on a diverse set of three publicly available datasets:
the MIMIC-CXR dataset, the CheXpert dataset, and the PadChest dataset. The
MIMIC-CXR dataset comprises an extensive collection of 377,110 frontal and
lateral chest radiographs from 227,835 radiological studies [36]. To ensure
the inclusion of unique patient information, we selected only the frontal
radiographs, specifically the anteroposterior (AP) or posteroanterior (PA)
views, for model pretraining, in cases where multiple radiographs were
available for the same patient. Within the MIMIC-CXR dataset, each radiograph
has been meticulously labeled using an NLP labeling tool [37] to indicate the
presence of 14 different pathological conditions. These conditions include
Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomediastinum,
Fracture, Lung Lesion, Lung Opacity, No Finding, Pleural Effusion, Pleural
Other, Pneumonia, Pneumothorax, and Support Devices.

The CheXpert dataset, a large and publicly available dataset, contributed
224,316 chest radiographs obtained from 65,240 patients [38]. Like the MIMIC-
CXR dataset, the radiographs in the official training split of CheXpert have
also been annotated using an NLP labeling tool [38] to identify the presence
of the same 14 pathological conditions as in the MIMIC-CXR dataset. In the
official test split of CheXpert, radiological ground truth was manually
annotated by five board-certified radiologists using majority vote [38].
Manually annotated CheXpert radiographs were used for evaluating the data
efficiency of our model, in section DiffChest is data efficient. The study
flow of CheXpert is shown in Figure S1.

Another dataset used in our study is PadChest, a publicly available dataset
comprising 160,861 chest radiographs from 67,625 patients at Hospital San
Juan, Spain [39]. The radiographs in PadChest were extensively annotated,
covering 193 different findings, including 174 distinct radiographic findings
and 19 differential diagnoses [39]. Notably, 27% of the images were manually
annotated by trained physicians, while the remaining 73% were automatically
labeled using a recurrent neural network [39]. We divided the PadChest dataset
into two subsets based on patients: The first subset, comprising 43,008
patients, contained only automatically labeled images, while the second
subset, consisting of 24,239 patients, contained only physician-labeled images
(see Figure 1). For evaluating the performance of our model, DiffChest, we
further partitioned 30% of patients with manually labeled studies from the
second subset to form a testing set, resulting in a total of 7,272 patients
dedicated to model evaluation.

### Image pre-processing

Each of the radiographs used in this study was resized to $256\times 256$ and
zero-padded before training and testing. In DiffChest, each image was then
normalized to the range of $[-1,1]$.

### Architecture

Our architecture comprises two main components: a self-conditioned diffusion
model, $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z})$, used for both
diffusion pretraining and image generation, and a feature extractor,
$p_{\phi}(\mathbf{z}|\mathbf{x}_{0})$. To facilitate image generation, we
adopted an enhanced convolutional U-Net architecture [40]. This U-Net
architecture can handle input resolutions of $256\times 256$. We leveraged the
efficacy of BigGAN residual blocks [41, 42] and global attention layers [43],
which were integrated into the U-Net at multiple resolution stages. These
additions enhance the model’s ability to capture intricate details and global
context, resulting in improved image generation. Adaptive group normalization
layers were used in the U-Net to facilitate self-conditioning [24, 40]. For
achieving self-conditioning, we made changes to the U-Net’s normalization
layers. Instead of using traditional normalization techniques like batch or
group normalization, we adopted adaptive group normalization layers [24]. The
feature extractor $p_{\phi}(\mathbf{z}|\mathbf{x}_{0})$ shares the same
architecture as the encoder part of the previously mentioned denoising U-Net.

### Implementation of DiffChest

Our self-supervised model consists of a denoising U-Net and a feature encoder
that we jointly train on 497,215 images. To prepare the data for pretraining,
all extracted images are normalized and stored in a single LMDB file. For
pretraining, we used an Adam optimizer with default $\beta_{1}$ = 0.9,
$\beta_{2}$ = 0.999, $\epsilon$ = 1e-8, and no weight decay to optimize the
diffusion loss (Equation 2). The training progress was measured by the number
of real images shown to DiffChest [35]. We trained our model with a fixed
learning rate of 1e-4 and a batch size of 12 until 200 million real
radiographs were shown to the model. Our classification head consisted of a
linear classifier, i.e., a logistic regression layer, which was trained on the
latent $\mathbf{z}$ space in all experiments. During classifier finetuning,
each latent vector was normalized using a sample mean and standard deviation
of the entire finetune dataset. All computations were performed on a GPU
cluster equipped with three Nvidia RTX A6000 48 GB GPUs (Nvidia, Santa Clara,
Calif). When not otherwise specified, the code implementations were in-house
developments based on Python 3.8 (https://www.python.org) and on the software
modules Numpy 1.24.3, Scipy 1.10.1, and Pytorch 2.0 [44].

### Model pretraining and finetuning

In total, our pretraining process used a dataset of 497,215 chest radiographs
from 170,717 patients. For the subsequent classification head finetuning, we
focused on a smaller set of 18,489 clinician-annotated radiographs sourced
from 16,967 patients. We used a single logistic regression layer to classify
the radiographs into 193 classes [39], see Extended Data Figure 1b.

### Model interpretability using visual explanations

Visual explanations are obtained by extrapolating the latent code from
$\mathbf{z}$ to $\mathbf{z}^{\ast}$ linearly, moving along the targeted
adversarial direction, as represented by Equation 1. The closed-form solution
for Equation 1 is given by:

$\mathbf{z}^{\ast}=\mathbf{z}+\epsilon\cdot\left(\mathbf{y}^{t}-\sigma(\mathbf{w}^{\intercal}\mathbf{z}+\mathbf{b})\right)\mathbf{w}^{\intercal},$
(4)

where $\mathbf{w}$ and $\mathbf{b}$ are the weight and bias of the logistic
regression layer, respectively, and $\sigma$ is the sigmoid function. The term
$\mathbf{y}^{t}\mathbf{w}^{\intercal}$ increases the probability of the target
class $y^{t}$, while the term
$-\sigma(\mathbf{w}^{\intercal}\mathbf{z}+\mathbf{b})\mathbf{w}^{\intercal}$
decreases the probability of the original class. To preserve the original
input information, we update only the latent code $\mathbf{z}$ towards the
target class, while keeping the original image attributes unchanged, i.e.,

$\mathbf{z}^{\ast}=\mathbf{z}+\epsilon\cdot\mathbf{y}^{t}\mathbf{w}^{\intercal}.$
(5)

In our experiments, we set the factor $\epsilon$ to 0.3. The manipulated
latent code $\mathbf{z}^{\ast}$ was then used to condition the reverse
diffusion process
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z}^{\ast})$, allowing us
to generate a new image $\mathbf{x}^{t}$. For this purpose, our conditional
diffusion model takes inputs $(\mathbf{z}^{\ast},\mathbf{x}_{T})$, where
$\mathbf{x}_{T}$ is an encoded noisy image used to initialize the diffusion
process (see section Input encoding). To expedite sample generation, we adopt
a non-Markovian DDIM sampler with 200 sampling steps proposed by Song et al.
[45].

### Design of reader study on confounder verification

Radiological confounders, as reconstructed by DiffChest, expose potential
biases that are specific to the data-collecting institutions. To validate the
accuracy and reliability of the identified confounding factors, we enlisted
the expertise of four radiologists. These experts carried out two tasks:
First, they annotated signals solely using data generated by DiffChest (as
depicted in Step 1 of Extended Data Figure 2); second, they verified these
confounding elements using the original radiographs (as shown in Step 3 of
Extended Data Figure 2). Separately, an independent machine-learning engineer
was brought in for Step 2 (refer to Extended Data Figure 2) to oversee the
data selection process and curate the test data set.

### Statistics

For each of the experiments, we calculated the ROC-AUC on the test set. If not
otherwise stated, we extracted 95% CI using bootstrapping with 1,000 redraws.
The difference in ROC-AUC was defined as the $\Delta$metric. To perform
bootstrapping, we built models for the total number of N = 10,000
bootstrapping by randomly permuting predictions of two classifiers, and then
computed metric differences $\Delta\text{metric}_{i}$ from their respective
scores. We obtained the two-tailed p-value of individual metrics by counting
all $\Delta\text{metric}_{i}$ above the threshold $\Delta$metric. Statistical
significance was defined as p < 0.001. For the clinical reader experiments, we
used Fleiss’ kappa to calculate inter-reader agreement between the three
radiologists.

## Data Availability

All datasets used in this study are publicly available: The chest radiography
datasets from MIMIC-CXR, CheXpert, and PadChest datasets can be requested from
the following URLs: MIMIC-CXR: https://physionet.org/content/mimic-cxr/2.0.0/
(requires credential access for potential users); CheXpert:
https://stanfordmlgroup.github.io/competitions/chexpert/; PadChest:
https://bimcv.cipf.es/bimcv-projects/padchest/. Testing labels and
radiologists’ annotations of CheXpert can be downloaded from
https://github.com/rajpurkarlab/cheXpert-test-set-labels.

## Code Availability

The code and pretrained models used in this study is made fully publicly
available under https://github.com/peterhan91/diffchest. Additionally, our
demo for generating visual explanations is accessible at
https://colab.research.google.com/drive/1gHWCQxreE1Olo2uQiXfSFSVInmiX85Nn?usp=sharing.

## Acknowledgements

This work is supported in part by the German Federal Ministry of Health (DEEP
LIVER, ZMVI1-2520DAT111) and the Max-Eder-Programme of the German Cancer Aid
70113864 (received by J.N.K).

## Author contributions

T.H., J.N.K., S.N., and D.T. devised the concept of the study, D.T., L.C.H.,
M.S.H. and R.M.S. performed the reader tests. T.H. wrote the code and
performed the performance studies. T.H. and D.T. did the statistical analysis.
T.H., J.N.K., and D.T. wrote the first draft of the manuscript. All authors
contributed to correcting the manuscript.

## Competing interests

J.N.K. declares consulting services for Owkin, France; DoMore Diagnostics,
Norway and Panakeia, UK; furthermore he holds shares in StratifAI GmbH and has
received honoraria for lectures by Bayer, Eisai, MSD, BMS, Roche, Pfizer and
Fresenius. D.T. holds shares in StratifAI GmbH. The other authors declare no
competing interests. The remaining authors declare no competing interests.

## References

  * [1] Castro, D. C., Walker, I. & Glocker, B.  Causality matters in medical imaging.  _Nature Communications_ 11, 3673 (2020).
  * [2] Glocker, B., Musolesi, M., Richens, J. & Uhler, C.  Causality in digital medicine.  _Nature Communications_ 12 (2021).
  * [3] Zeng, J., Gensheimer, M. F., Rubin, D. L., Athey, S. & Shachter, R. D.  Uncovering interpretable potential confounders in electronic medical records.  _Nature Communications_ 13, 1014 (2022).
  * [4] Mukherjee, P. _et al._ Confounding factors need to be accounted for in assessing bias by machine learning algorithms.  _Nature Medicine_ 28, 1159–1160 (2022).
  * [5] Rueckel, J. _et al._ Impact of confounding thoracic tubes and pleural dehiscence extent on artificial intelligence pneumothorax detection in chest radiographs.  _Investigative Radiology_ 55, 792–798 (2020).
  * [6] Zhao, Q., Adeli, E. & Pohl, K. M.  Training confounder-free deep learning models for medical applications.  _Nature communications_ 11, 6010 (2020).
  * [7] De Sousa Ribeiro, F., Xia, T., Monteiro, M., Pawlowski, N. & Glocker, B.  High fidelity image counterfactuals with probabilistic causal models.  _arXiv e-prints_ arXiv–2306 (2023).
  * [8] Zech, J. R. _et al._ Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.  _PLoS medicine_ 15, e1002683 (2018).
  * [9] DeGrave, A. J., Janizek, J. D. & Lee, S.-I.  Ai for radiographic covid-19 detection selects shortcuts over signal.  _Nature Machine Intelligence_ 3, 610–619 (2021).
  * [10] Ho, J., Jain, A. & Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33, 6840–6851 (2020).
  * [11] Ho, J. _et al._ Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ (2022).
  * [12] Li, X., Thickstun, J., Gulrajani, I., Liang, P. S. & Hashimoto, T. B.  Diffusion-lm improves controllable text generation.  _Advances in Neural Information Processing Systems_ 35, 4328–4343 (2022).
  * [13] Khader, F. _et al._ Denoising diffusion probabilistic models for 3d medical image generation.  _Scientific Reports_ 13, 7303 (2023).
  * [14] Moor, M. _et al._ Foundation models for generalist medical artificial intelligence.  _Nature_ 616, 259–265 (2023).
  * [15] Tiu, E. _et al._ Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning.  _Nature Biomedical Engineering_ 6, 1399–1406 (2022).
  * [16] Kundu, S.  Ai in medicine must be explainable.  _Nature medicine_ 27, 1328–1328 (2021).
  * [17] Sundararajan, M., Taly, A. & Yan, Q.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , 3319–3328 (PMLR, 2017).
  * [18] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. & Torralba, A.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2921–2929 (2016).
  * [19] Selvaraju, R. R. _et al._ Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626 (2017).
  * [20] Han, T. _et al._ Advancing diagnostic performance and clinical usability of neural networks via adversarial training and dual batch normalization.  _Nature communications_ 12, 4315 (2021).
  * [21] Han, T. _et al._ Image prediction of disease progression for osteoarthritis by style-based manifold extrapolation.  _Nature Machine Intelligence_ 4, 1029–1039 (2022).
  * [22] Kim, G., Kwon, T. & Ye, J. C.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2426–2435 (2022).
  * [23] Kwon, M., Jeong, J. & Uh, Y.  Diffusion models already have a semantic latent space.  _arXiv preprint arXiv:2210.10960_ (2022).
  * [24] Preechakul, K., Chatthee, N., Wizadwongsa, S. & Suwajanakorn, S.  Diffusion autoencoders: Toward a meaningful and decodable representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10619–10629 (2022).
  * [25] Barber, D. & Agakov, F.  The im algorithm: a variational approach to information maximization.  _Advances in neural information processing systems_ 16, 201 (2004).
  * [26] Chen, X. _et al._ Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  _Advances in neural information processing systems_ 29 (2016).
  * [27] Seyyed-Kalantari, L., Zhang, H., McDermott, M. B., Chen, I. Y. & Ghassemi, M.  Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations.  _Nature medicine_ 27, 2176–2182 (2021).
  * [28] Azizi, S. _et al._ Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging.  _Nature Biomedical Engineering_ 1–24 (2023).
  * [29] Oord, A. v. d., Li, Y. & Vinyals, O.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * [30] He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 9729–9738 (2020).
  * [31] Chen, T., Kornblith, S., Norouzi, M. & Hinton, G.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607 (PMLR, 2020).
  * [32] Bachman, P., Hjelm, R. D. & Buchwalter, W.  Learning representations by maximizing mutual information across views.  _Advances in neural information processing systems_ 32 (2019).
  * [33] Nguyen, T. T. _et al._ Deep learning for deepfakes creation and detection: A survey.  _Computer Vision and Image Understanding_ 223, 103525 (2022).
  * [34] Ho, J. _et al._ Cascaded diffusion models for high fidelity image generation.  _The Journal of Machine Learning Research_ 23, 2249–2281 (2022).
  * [35] Karras, T., Aila, T., Laine, S. & Lehtinen, J.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ (2017).
  * [36] Johnson, A. E. _et al._ Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  _Scientific data_ 6, 317 (2019).
  * [37] Peng, Y. _et al._ Negbio: a high-performance tool for negation and uncertainty detection in radiology reports.  _AMIA Summits on Translational Science Proceedings_ 2018, 188 (2018).
  * [38] Irvin, J. _et al._ Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, 590–597 (2019).
  * [39] Bustos, A., Pertusa, A., Salinas, J.-M. & De La Iglesia-Vaya, M.  Padchest: A large chest x-ray image dataset with multi-label annotated reports.  _Medical image analysis_ 66, 101797 (2020).
  * [40] Dhariwal, P. & Nichol, A.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ 34, 8780–8794 (2021).
  * [41] Ronneberger, O., Fischer, P. & Brox, T.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , 234–241 (Springer, 2015).
  * [42] Brock, A., Donahue, J. & Simonyan, K.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ (2018).
  * [43] Wang, X., Girshick, R., Gupta, A. & He, K.  Non-local neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7794–7803 (2018).
  * [44] Paszke, A. _et al._ Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * [45] Song, J., Meng, C. & Ermon, S.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ (2020).
  * [46] Lu, M. Y. _et al._ Data-efficient and weakly supervised computational pathology on whole-slide images.  _Nature biomedical engineering_ 5, 555–570 (2021).

Extended Data Fig. 1: Schematic of DiffChest workflows. (a) DiffChest starts
with self-supervised pretraining on an unlabeled dataset. In contrast to
current contrastive methods, DiffChest relies on the simplified diffusion
objective (Equation 2) to jointly train a feature encoder and a denoising
diffusion model. (b) It involves supervised fine-tuning on a labeled subset.
We only train a logistic regression classifier on top of the pretrained
encoder and keep the encoder weights fixed. (c) Our method is both
discriminative and generative. We use encoder-extracted features to train a
classifier and employ the diffusion model to generate visual explanations for
prediction classes. The latter is particularly crucial for clinicians to
accept AI-assisted diagnosis, where interpretability is critical. We perturb
the latent feature towards a target class and utilize the diffusion model to
generate a visual explanation for the target class while preserving the
original information (Equation 5).

Extended Data Fig. 2: Design of confounder reader study. The purpose of this
reader study is to validate the clinical relevance of the confounding signals
visualized by our visual explanations (Figure 4). Unlike previous studies [9],
our model is capable of discovering possible bias signals for multiple
clinically relevant predictions. For instance, a confident diagnosis made by
an AI model on cardiopulmonary disease may not solely be based on disease
features but rather due to the presence of supporting devices or sternum
bindings. To verify if the observed confounders indeed represent biases in the
training data, we invited four experienced radiologists to review the visual
explanations generated by our model (step 1), and rate the existence of the
confounding signals in the training data (step 3).

Extended Data Fig. 3: Confounders identified by a radiologist using visual
explanations from DiffChest. This figure is the result of step-1 suggested in
Extended Data Figure 2. Both treatment a and physiology-related b confounders
are identified by a radiologist using visual explanations from DiffChest.

Extended Data Fig. 4: Inter-reader agreement on treatment-related confounders.
Each cell represents the number of radiographs that are rated as having the
corresponding confounder by both radiologists.

Extended Data Fig. 5: Inter-reader agreement on physiology-related
confounders. Each cell represents the number of radiographs that are rated as
having the corresponding confounder by both radiologists.

## Supplementary Information

##  A Extended derivations

### Mutual information

Mutual information is a general way to measure dependency between two random
variables. In information theory, the mutual information can be defined as the
difference of two entropies:

$I(\mathbf{z};\mathbf{x})=H(\mathbf{z})-H(\mathbf{z}|\mathbf{x})=H(\mathbf{x})-H(\mathbf{x}|\mathbf{z}).$
(6)

$I(\mathbf{z};\mathbf{x})$ is the reduction of uncertainty in $\mathbf{x}$
when $\mathbf{z}$ is observed. In self-supervised learning, we hope to find a
compressed and informative representation of the input data, i.e., maximizing
the mutual information between the input and the latent representation.
However, it is hard to compute the mutual information directly due to the
intractable posterior:

$\displaystyle I(\mathbf{z};\mathbf{x})$
$\displaystyle=H(\mathbf{z})-H(\mathbf{z}|\mathbf{x})$ (7)
$\displaystyle=H(\mathbf{z})-\mathbb{E}_{(\mathbf{z,x})\sim
p(\mathbf{z,x})}\left[-\log p(\mathbf{z}|\mathbf{x})\right].$

In VIM [26], we can estimate the posterior by introducing a variational
distribution $p_{\phi}(\mathbf{z}|\mathbf{x})$:

$\displaystyle I(\mathbf{z};\mathbf{x})$
$\displaystyle=H(\mathbf{z})-\mathbb{E}_{(\mathbf{z,x})\sim
p(\mathbf{z,x})}\left[-\log p(\mathbf{z}|\mathbf{x})\right]$ (8)
$\displaystyle=H(\mathbf{z})+\mathbb{E}_{(\mathbf{z,x})\sim
p(\mathbf{z,x})}\left[\log p(\mathbf{z}|\mathbf{x})-\log
p_{\phi}(\mathbf{z}|\mathbf{x})+\log p_{\phi}(\mathbf{z}|\mathbf{x})\right]$
$\displaystyle=H(\mathbf{z})+\mathbb{E}_{(\mathbf{z,x})\sim
p(\mathbf{z,x})}\left[D_{\text{KL}}\left(p(\mathbf{z}|\mathbf{x})\,||\,p_{\theta}(\mathbf{z}|\mathbf{x})\right)+\log
p_{\phi}(\mathbf{z}|\mathbf{x})\right]$ $\displaystyle\geq
H(\mathbf{z})+\mathbb{E}_{(\mathbf{z,x})\sim p(\mathbf{z,x})}\left[\log
p_{\phi}(\mathbf{z}|\mathbf{x})\right].$

The entropy $H(\mathbf{z})$ is typically treated as a constant, and the lower
bound of mutual information is given by $\mathbb{E}_{(\mathbf{z,x})\sim
p(\mathbf{z,x})}\left[\log p_{\phi}(\mathbf{z}|\mathbf{x})\right]$ (Equation
3).

### Simplified diffusion objective

Next, we will derive the simplified diffusion objective from the variational
lower bound (VLB) on negative log joint likelihood $-\log
p_{\theta,\phi}(\mathbf{x}_{0},\mathbf{z})$ in DiffChest. Given the forward
diffusion process as $q\left(\mathbf{x}_{1:T}|\mathbf{x}_{0}\right)$ and our
designed backward process as
$p_{\theta,\phi}\left(\mathbf{x}_{1:T}|\mathbf{x}_{0},\mathbf{z}\right)$, we
can derive the VLB as:

$\displaystyle-\log p_{\theta,\phi}(\mathbf{x}_{0},\mathbf{z})$
$\displaystyle\leq-\log
p_{\theta,\phi}(\mathbf{x}_{0},\mathbf{z})+D_{\text{KL}}\left(q\left(\mathbf{x}_{1:T}|\mathbf{x}_{0}\right)||\;p_{\theta,\phi}\left(\mathbf{x}_{1:T}|\mathbf{x}_{0},\mathbf{z}\right)\right)$
(9) $\displaystyle=-\log
p_{\theta,\phi}(\mathbf{x}_{0},\mathbf{z})+\mathbb{E}_{\mathbf{x}_{1:T}\sim
q\left(\mathbf{x}_{1:T}|\mathbf{x}_{0}\right)}\left[\log\frac{q\left(\mathbf{x}_{1:T}|\mathbf{x}_{0}\right)}{p_{\theta,\phi}\left(\mathbf{x}_{1:T}|\mathbf{x}_{0},\mathbf{z}\right)}\right]$
$\displaystyle=\mathbb{E}_{(\mathbf{x}_{0:T},\mathbf{z})\sim
q\left(\mathbf{x}_{0:T},\mathbf{z}\right)}\left[\log\frac{q\left(\mathbf{x}_{1:T}|\mathbf{x}_{0}\right)}{p_{\theta,\phi}\left(\mathbf{x}_{0:T},\mathbf{z}\right)}\right]$
$\displaystyle=\mathbb{E}_{q}\left[\log\frac{\prod_{t=1}^{T}q(\mathbf{x}_{t}|\mathbf{x}_{t-1})}{p(\mathbf{x}_{T},\mathbf{z})\prod_{t=1}^{T}p_{\theta,\phi}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z})}\right]$
$\displaystyle=\mathbb{E}_{q}\left[-\log
p(\mathbf{x}_{T},\mathbf{z})+\sum_{t=1}^{T}\log\frac{q(\mathbf{x}_{t}|\mathbf{x}_{t-1})}{p_{\theta,\phi}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z})}\right]$
$\displaystyle=\mathbb{E}_{q}\left[-\log
p(\mathbf{x}_{T},\mathbf{z})+\sum_{t=2}^{T}\log\frac{q(\mathbf{x}_{t}|\mathbf{x}_{t-1})}{p_{\theta,\phi}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z})}+\log\frac{q(\mathbf{x}_{1}|\mathbf{x}_{0})}{p_{\theta,\phi}(\mathbf{x}_{0}|\mathbf{x}_{1},\mathbf{z})}\right]$
$\displaystyle=\mathbb{E}_{q}\left[-\log
p(\mathbf{x}_{T},\mathbf{z})+\sum_{t=2}^{T}\log\left(\frac{q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})}{p_{\theta,\phi}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z})}\cdot\frac{q(\mathbf{x}_{t}|\mathbf{x}_{0})}{q(\mathbf{x}_{t-1}|\mathbf{x}_{0})}\right)+\log\frac{q(\mathbf{x}_{1}|\mathbf{x}_{0})}{p_{\theta,\phi}(\mathbf{x}_{0}|\mathbf{x}_{1},\mathbf{z})}\right]$
$\displaystyle=\mathbb{E}_{q}\left[\underbrace{\log\frac{q(\mathbf{x}_{T}|\mathbf{x}_{0})}{p(\mathbf{x}_{T},\mathbf{z})}\vphantom{\sum_{t=2}^{T}}}_{L_{T}}+\underbrace{\sum_{t=2}^{T}\log\frac{q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})}{p_{\theta,\phi}(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{z})}}_{L_{t}}-\underbrace{\log
p_{\theta,\phi}(\mathbf{x}_{0}|\mathbf{x}_{1},\mathbf{z})\vphantom{\sum_{t=2}^{T}}}_{L_{0}}\right].$

The VLB is now factorized into three terms, namely $L_{0}$, $L_{t}$, and
$L_{T}$. We may ignore $L_{T}$ since it has no learnable parameters and
$\mathbf{x}_{T}$ is by definition a Gaussian noise. Following, [24], we model
$L_{0}$ using a deterministic decoder derived from
$\mathcal{N}(\mathbf{x}_{0};\mu_{\theta,\phi}(\mathbf{x}_{1},1,\mathbf{z}),\mathbf{0})$.
In DDIM’s formulation, the mean of the reverse kernel
$\mu_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})$ is parameterized as

$\mu_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\sqrt{1-\alpha_{t}}\,\mathbf{\epsilon}_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})\right),$
(10)

where $\alpha_{t}$ is the diffusion parameter used in [45].

Equation 2 is derived from $L_{t}$. Recall, Ho et al. proved the forward
kernel $q(x_{t-1}|x_{t},x_{0})$ follows Gaussian distributions [10]. The
backward kernel $p_{\theta,\phi}(x_{t-1}|x_{t},z)$ is also Gaussian since the
step size is small. $L_{t}$ is
$D_{\text{KL}}\left(q(x_{t-1}|x_{t},x_{0})\;||\;p_{\theta,\phi}(x_{t-1}|x_{t},z)\right)$,
since both distributions are Gaussian, we can derive the closed-form solution
for $L_{t}$:

$\displaystyle L_{t}$
$\displaystyle=\mathbb{E}_{\mathbf{x}_{0},\mathbf{\epsilon}}\left[\frac{1}{2\left\|\Sigma_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})\right\|_{2}^{2}}\left\|\mu_{t}(\mathbf{x}_{t},\mathbf{x}_{0})-\mu_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})\right\|^{2}\right]$
(11)
$\displaystyle=\mathbb{E}_{\mathbf{x}_{0},\mathbf{\epsilon}}\left[\frac{(1-\alpha_{t})^{2}}{2\alpha_{t}(1-\bar{\alpha}_{t})\left\|\Sigma_{\theta,\phi}\right\|_{2}^{2}}\left\|\mathbf{\epsilon}_{t}-\mathbf{\epsilon}_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})\right\|^{2}\right],$

where $\Sigma_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})$ is the variance and
$\mathbf{\epsilon}_{t}$ is the Gaussian noise added at step $t$. We can get
the simplified diffusion objective (Equation 2) by neglecting the weighting
term in Equation 11.

### Mutual information maximized pretraining

We have proven in the previous section that optimizing our diffusion
pretraining loss, i.e., Equation 2, is equivalent to maximizing the VLB of the
likelihood $\log p_{\theta,\phi}(\mathbf{x}_{0},\mathbf{z})=\log
p_{\theta}(\mathbf{x}_{0})+\log p_{\theta,\phi}(\mathbf{z}|\mathbf{x}_{0})$.
The first term, $\log p_{\theta}(\mathbf{x}_{0})$, represents the log
likelihood of the input data, while the second term, $\log
p_{\theta,\phi}(\mathbf{z}|\mathbf{x}_{0})$, corresponds to the VLB of the
mutual information between the input and the latent representation (Equation
8). Therefore, optimizing our diffusion pretraining loss is equivalent to
maximizing both the data likelihood and the mutual information between the
input $\mathbf{x}$ and its latent representation $\mathbf{z}$.

##  B Visual explanation generation

### Input encoding

The latent space of DiffChest consists of two parts: a 512-dimensional latent
code $\mathbf{z}$ and a 256$\times$256 noise map $\mathbf{x}_{T}$, as depicted
in Extended Data Figure 1. To generate visual explanations, we first need to
encode the input image $\mathbf{x}_{0}$ into both latent spaces. Embedding
$\mathbf{x}_{0}$ into $\mathbf{z}$ space is accomplished using our trained
feature extractor $p_{\phi}(\mathbf{z}|\mathbf{x}_{0})$. Similarly, we encode
$\mathbf{x}_{0}$ into $\mathbf{x}_{T}$ space by running the generative process
in our diffusion model backward. This process is deterministic as we utilized
a DDIM sampler for sample generation:

$\mathbf{x}_{t}=\sqrt{\alpha_{t}}\,\mu_{\theta,\phi}(\mathbf{x_{t-1}},t-1,\mathbf{z})+\sqrt{1-\alpha_{t}}\,\mathbf{\epsilon}_{\theta,\phi}(\mathbf{x}_{t-1},t-1,\mathbf{z}).$
(12)

Here, the mean $\mu_{\theta,\phi}(\mathbf{x}_{t},t,\mathbf{z})$ is defined in
Equation 10. In our experiments, we set the number of encoding steps to 250.
After encoding, we can use both the encoded $\mathbf{z}$ and $\mathbf{x}_{T}$
(as shown in Figure S6) to reconstruct the input image. As depicted in Figure
S6, sharp and high-fidelity reconstructions can be achieved by using our
DiffChest with just 100 sampling steps.

##  C DiffChest is data efficient

Learning from few annotated datasets poses a significant challenge in medical
AI, as the predictive performance of models generally improves with larger
training datasets [46] and as high-quality medical data for training purposes
is scarce. To address this issue, we investigated the potential of diffusion
pretraining to mitigate performance losses when using only a small subset of
radiographs from the CheXpert dataset for model tuning (the data flow is given
in Figure S1). We employed CheXpert for our study due to the availability of
high-quality testing labels and annotations from three board-certified
radiologists [38]. Our findings demonstrate that DiffChest’s performance
remains stable and almost on par with radiologists when finetuning the model
using only 10% of the total data (Figure S2). However, it is worth noting that
all radiologists clearly outperformed DiffChest on the classification of
supporting devices. We attribute this to the presence of label noise within
the CheXpert dataset, as DiffChest achieved an almost perfect AUC when
classifying device labels such as single chamber device, dual chamber device,
and pacemaker in the PadChest dataset (Figure 2). Even when downsampled to
only 3% of the total data, DiffChest maintains high AUC scores of at least
0.800 for No Finding, Edema, Supporting Devices, and Pleural Effusion.

##  D Supplementary tables and figures

Extended Data Fig. S1: Performance comparison between DiffChest and CheXzero. Patient subgroup | Diffchest (95% CI) | CheXzero (95% CI) | p-value
---|---|---|---
sternotomy | 0.988 (0.982, 0.994) | 0.810 (0.772, 0.847) | <1e-4
endotracheal tube | 0.978 (0.968, 0.986) | 0.987 (0.983, 0.990) | 0.820
pacemaker | 0.978 (0.967, 0.989) | 0.992 (0.989, 0.995) | 0.660
nsg tube | 0.977 (0.970, 0.983) | 0.928 (0.909, 0.949) | 0.147
dual chamber device | 0.977 (0.968, 0.985) | 0.974 (0.967, 0.979) | 0.960
central venous catheter via jugular vein | 0.964 (0.944, 0.982) | 0.975 (0.966, 0.984) | 0.823
single chamber device | 0.946 (0.927, 0.967) | 0.973 (0.966, 0.979) | 0.684
mammary prosthesis | 0.944 (0.903, 0.987) | 0.532 (0.444, 0.630) | <1e-4
pleural effusion | 0.943 (0.933, 0.954) | 0.971 (0.964, 0.978) | 0.195
tracheostomy tube | 0.940 (0.894, 0.983) | 0.960 (0.933, 0.986) | 0.724
heart insufficiency | 0.936 (0.922, 0.951) | 0.936 (0.920, 0.951) | 0.992
cardiomegaly | 0.899 (0.890, 0.909) | 0.903 (0.894, 0.911) | 0.795
alveolar pattern | 0.894 (0.874, 0.913) | 0.898 (0.877, 0.918) | 0.862
hiatal hernia | 0.893 (0.864, 0.924) | 0.826 (0.792, 0.861) | 0.083
hilar congestion | 0.885 (0.855, 0.914) | 0.903 (0.882, 0.925) | 0.651
goiter | 0.874 (0.830, 0.920) | 0.605 (0.542, 0.672) | <1e-4
gynecomastia | 0.872 (0.842, 0.904) | 0.563 (0.508, 0.614) | <1e-4
central venous catheter via subclavian vein | 0.872 (0.824, 0.919) | 0.917 (0.889, 0.946) | 0.414
kyphosis | 0.869 (0.844, 0.891) | 0.722 (0.690, 0.753) | <1e-4
aortic elongation | 0.865 (0.852, 0.878) | 0.675 (0.658, 0.693) | <1e-4
interstitial pattern | 0.850 (0.830, 0.868) | 0.843 (0.822, 0.863) | 0.739
osteosynthesis material | 0.840 (0.770, 0.917) | 0.576 (0.495, 0.652) | <1e-4
emphysema | 0.829 (0.786, 0.873) | 0.801 (0.749, 0.854) | 0.573
suboptimal study | 0.827 (0.795, 0.858) | 0.783 (0.747, 0.822) | 0.197
normal | 0.825 (0.816, 0.834) | 0.752 (0.741, 0.763) | <1e-4
apical pleural thickening | 0.823 (0.802, 0.844) | 0.641 (0.611, 0.673) | <1e-4
descendent aortic elongation | 0.819 (0.757, 0.882) | 0.789 (0.745, 0.838) | 0.647
hypoexpansion basal | 0.815 (0.737, 0.892) | 0.806 (0.753, 0.859) | 0.916
copd signs | 0.814 (0.800, 0.828) | 0.755 (0.738, 0.772) | <1e-4
costophrenic angle blunting | 0.808 (0.785, 0.831) | 0.798 (0.775, 0.823) | 0.636
aortic atheromatosis | 0.805 (0.784, 0.824) | 0.699 (0.675, 0.723) | <1e-4
bullas | 0.805 (0.729, 0.878) | 0.557 (0.464, 0.653) | <1e-4
volume loss | 0.804 (0.769, 0.840) | 0.730 (0.692, 0.770) | 0.027
osteopenia | 0.798 (0.725, 0.869) | 0.673 (0.602, 0.749) | 0.051
atelectasis | 0.792 (0.753, 0.831) | 0.814 (0.781, 0.851) | 0.507
tracheal shift | 0.791 (0.700, 0.871) | 0.699 (0.630, 0.765) | 0.137
pleural thickening | 0.790 (0.715, 0.867) | 0.822 (0.768, 0.878) | 0.601
chronic changes | 0.789 (0.775, 0.804) | 0.654 (0.636, 0.671) | <1e-4
vertebral degenerative changes | 0.779 (0.759, 0.800) | 0.644 (0.616, 0.672) | <1e-4
azygos lobe | 0.779 (0.706, 0.852) | 0.465 (0.352, 0.579) | <1e-4
tuberculosis sequelae | 0.778 (0.696, 0.857) | 0.642 (0.557, 0.725) | 0.032
diaphragmatic eventration | 0.776 (0.730, 0.825) | 0.602 (0.547, 0.652) | <1e-4
pneumonia | 0.770 (0.745, 0.796) | 0.817 (0.793, 0.841) | 0.038
air trapping | 0.765 (0.740, 0.791) | 0.590 (0.561, 0.618) | <1e-4
bronchovascular markings | 0.764 (0.713, 0.818) | 0.686 (0.640, 0.730) | 0.038
hemidiaphragm elevation | 0.761 (0.724, 0.797) | 0.847 (0.817, 0.877) | 0.010
scoliosis | 0.753 (0.731, 0.774) | 0.646 (0.623, 0.672) | <1e-4
bronchiectasis | 0.752 (0.708, 0.795) | 0.751 (0.710, 0.794) | 0.966
vertebral anterior compression | 0.745 (0.701, 0.791) | 0.636 (0.586, 0.686) | 0.006
suture material | 0.745 (0.690, 0.800) | 0.558 (0.503, 0.613) | <1e-4
callus rib fracture | 0.743 (0.706, 0.781) | 0.659 (0.621, 0.697) | 0.008
pulmonary mass | 0.740 (0.658, 0.818) | 0.904 (0.864, 0.945) | 0.008
consolidation | 0.734 (0.659, 0.806) | 0.888 (0.846, 0.928) | 0.004
infiltrates | 0.720 (0.689, 0.752) | 0.772 (0.747, 0.800) | 0.044
laminar atelectasis | 0.717 (0.684, 0.751) | 0.689 (0.657, 0.720) | 0.291
nipple shadow | 0.715 (0.658, 0.771) | 0.633 (0.574, 0.688) | 0.077
vascular hilar enlargement | 0.714 (0.685, 0.745) | 0.631 (0.598, 0.665) | 0.002
fibrotic band | 0.712 (0.682, 0.748) | 0.633 (0.601, 0.665) | 0.003
aortic button enlargement | 0.704 (0.631, 0.781) | 0.700 (0.639, 0.762) | 0.962
lobar atelectasis | 0.690 (0.596, 0.781) | 0.807 (0.734, 0.879) | 0.099
hilar enlargement | 0.686 (0.634, 0.738) | 0.679 (0.623, 0.735) | 0.852
metal | 0.676 (0.598, 0.751) | 0.536 (0.462, 0.610) | 0.007
hyperinflated lung | 0.673 (0.589, 0.760) | 0.748 (0.664, 0.829) | 0.257
flattened diaphragm | 0.662 (0.583, 0.746) | 0.567 (0.476, 0.660) | 0.172
rib fracture | 0.650 (0.541, 0.752) | 0.771 (0.689, 0.855) | 0.093
unchanged | 0.648 (0.629, 0.667) | 0.544 (0.524, 0.566) | <1e-4
pseudonodule | 0.635 (0.591, 0.676) | 0.510 (0.468, 0.549) | <1e-4
calcified granuloma | 0.632 (0.589, 0.678) | 0.471 (0.423, 0.518) | <1e-4
granuloma | 0.619 (0.535, 0.700) | 0.476 (0.382, 0.563) | 0.040
calcified densities | 0.618 (0.555, 0.683) | 0.528 (0.471, 0.593) | 0.065
ground glass pattern | 0.598 (0.494, 0.717) | 0.655 (0.553, 0.760) | 0.427
nodule | 0.587 (0.541, 0.635) | 0.580 (0.531, 0.630) | 0.854
increased density | 0.567 (0.525, 0.608) | 0.680 (0.638, 0.719) | <1e-4

Figure S1: Data flow chart for CheXpert dataset. Figure S2: DiffChest is data
efficient. After completing the diffusion pretraining, we compared three
models: classification heads finetuned using 100% ($N=7,943$), 10% ($N=794$),
and 3% ($N=238$) of the CheXpert training data. To evaluate and compare the
performance, we used receiver operating characteristic (ROC) curves for these
models against benchmarking radiologists provided by the CheXpert competition.
The testing cohort consisted of $N=500$ radiographs.  Figure S3: Visual
explanations from DiffChest for the 1st to the 24th clinical finding
predictions shown in Figure 2. We visualize the original image in the top left
panel of this image grid.  Figure S4: Visual explanations from DiffChest for
the 25th to the 48th clinical finding predictions shown in Figure 2. We
visualize the original image in the top left panel of this image grid.  Figure
S5: Visual explanations from DiffChest for the 49th to the 72th clinical
finding predictions shown in Figure 2. We visualize the original image in the
top left panel of this image grid.  Figure S6: Encoding input images to
diffusion latent space. We embed the input images into the diffusion latent
space following a deterministic multistep process (Equation 12). The top row
shows the original images, and the bottom row shows the corresponding latent
codes $\mathbf{x}_{T}$. In the middle, we visualize reconstructions of the
original images from the latent codes.


# Growth and decay of Hölder moduli111AMS Subject Classification (2020): 42B25
(primary); 42C15, 42C40 (secondary). Key words: vaguelets, wavelets, almost-
orthogonality.

Michael Wilson
Department of Mathematics
University of Vermont
Burlington, Vermont 05405
<EMAIL_ADDRESS>
###### Abstract

If $f:{{\bf R}^{d}}\to{\bf C}$ is bounded and $f$’s Hölder $\alpha$-modulus of
continuity grows no faster than $(1+|x|)^{M}$ ($M\geq 0$) then, for every
$\epsilon>0$, there is a $\beta>0$ such that $f$’s Hölder $\beta$-modulus
grows no faster than $(1+|x|)^{\epsilon}$. We use this easy fact to show that,
if $|f|$ decays as fast as $(1+|x|)^{-R}$ (for $R>0$) and $f$’s
$\alpha$-Hölder modulus grows no faster than $(1+|x|)^{M}$, then, for every
$0\leq R^{\prime}<R$, there is a $\beta>0$ such that $f$’s $\beta$-Hölder
modulus decays as fast as $(1+|x|)^{-R^{\prime}}$. We apply this to strengthen
a result of Coifman and Meyer on almost-orthogonality of vaguelet families and
to derive other useful facts about vaguelets and vaguelet-like functions.

In [3, p. 56, Theorem 2] Coifman and Meyer proved

###### Theorem 1

Let $0<\alpha<\epsilon$ and $C_{1},C_{2}>0$. Let $\\{f^{(Q)}\\}_{Q\in{\cal
D}_{d}}$ be a family of functions indexed over the dyadic cubes ${\cal D}_{d}$
in ${{\bf R}^{d}}$, where each $f^{(Q)}$ has integral 0 and: i) for all
$x\in{{\bf R}^{d}}$,

$|f^{(Q)}(x)|\leq C_{1}(1+|x|)^{-d-\epsilon};$ (1)

ii) for all $x,x^{\prime}\in{{\bf R}^{d}}$,

$|f^{(Q)}(x)-f^{(Q)}(x^{\prime})|\leq C_{2}|x-x^{\prime}|^{\alpha}.$ (2)

For each $Q\in{\cal D}_{d}$, let $x_{Q}$ be $Q$’s center and $\ell(Q)$ its
sidelength, and define

$g^{(Q)}(x):={f^{(Q)}((x-x_{Q})/\ell(Q))\over|Q|^{1/2}},$

where $|Q|$ means $Q$’s Lebesgue measure. There is a constant $A$, depending
only on $\epsilon$, $\alpha$, $C_{1}$, $C_{2}$, and $d$, so that, for all
finite sequences $\\{\lambda_{Q}\\}_{Q\in{\cal D}_{d}}\subset{\bf C}$,

$\int\left|\sum\lambda_{Q}g^{(Q)}\right|^{2}\,dx\leq A\sum|\lambda_{Q}|^{2}.$
(3)

_Remark_. The hypothesis $\alpha<\epsilon$ is unnecessary (see Lemma 1 below).

Coifman and Meyer call the functions $g^{(Q)}$ _vaguelets_. Theorem 1 shows
that $\\{g^{(Q)}\\}_{Q\in{\cal D}_{d}}$ is a _Bessel family_ ; property (3) is
sometimes called _almost-orthogonality_. Theorem 1 is well known. Vaguelets
are familiar objects in frame theory and signal processing ([1][2][4][5][6])
if for no other reason than that they are what you get when you apply singular
integral operators to wavelets.

In this paper we prove

###### Theorem 2

Let $D_{1},R,\alpha$ be positive numbers and $M\geq 0$. For every $0\leq
R^{\prime}<R$ there are positive constants $D_{2}$ and $\beta$, depending only
on $D_{1},\alpha,R,R^{\prime}$, and $M$, such that the following holds: If
$f:{{\bf R}^{d}}\to{\bf C}$ satisfies

$|f(x)|\leq(1+|x|)^{-R}$ (4)

for all $x\in{{\bf R}^{d}}$ and

$|f(x)-f(x^{\prime})|\leq
D_{1}|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{M}$ (5)

for all $x,x^{\prime}\in{{\bf R}^{d}}$, then

$|f(x)-f(x^{\prime})|\leq
D_{2}|x-x^{\prime}|^{\beta}(1+\min(|x|,|x^{\prime}|))^{-R^{\prime}}$ (6)

for all $x,x^{\prime}\in{{\bf R}^{d}}$.

_Remark_. The switch from ‘max’ in (5) to ‘min’ in (6) is not a typo. The
theorem says: If $f$ decays as fast as $(1+|x|)^{-R}$ and its $\alpha$-Hölder
modulus grows no faster than a polynomial then, for some $\beta>0$, $f$’s
$\beta$-Hölder modulus decays almost as fast as $(1+|x|)^{-R}$. The relevance
of this to Theorem 1 is as follows. Put $R:=d+\epsilon$ and suppose that every
$f^{(Q)}$ in $\\{f^{(Q)}\\}_{Q\in{\cal D}_{d}}$ has integral 0 and satisfies
(4) and (5). By Theorem 2, every $f^{(Q)}$ satisfies (4) for this $R$ and (6)
for $R^{\prime}:=0$ and some $\beta>0$. Theorem 1 follows by our preceding
remark. Theorem 2 also has this nice corollary.

###### Corollary 1

Let $\epsilon,C_{1},C_{2}>0$ and $M\geq 0$. Let $\\{f^{(Q)}\\}_{Q\in{\cal
D}_{d}}$ be a family of differentiable functions indexed over the dyadic cubes
${\cal D}_{d}$ in ${{\bf R}^{d}}$, where each $f^{(Q)}$ has integral 0 and,
for all $x\in{{\bf R}^{d}}$,

$|f^{(Q)}(x)|\leq C_{1}(1+|x|)^{-d-\epsilon}$

and

$|\nabla f^{(Q)}(x)|\leq C_{2}(1+|x|)^{M}.$

If the functions $g^{(Q)}$ are defined as in Theorem 1 then (3) holds for some
$A$ depending only on $\epsilon,C_{1},C_{2},M$, and $d$.

Theorem 2 follows from a simple lemma which may be of independent interest. We
believe it’s well known, but we haven’t seen it anywhere.

###### Lemma 1

Let $A,\alpha,M\geq 0$ and let $f:{{\bf R}^{d}}\to{\bf C}$ satisfy: a) for all
$x\in{{\bf R}^{d}}$, $|f(x)|\leq 1$; b) for all $x,x^{\prime}\in{{\bf
R}^{d}}$,

$|f(x)-f(x^{\prime})|\leq
A|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{M}.$

Then, for all $0<\rho\leq 1$ and $x,x^{\prime}\in{{\bf R}^{d}}$,

$|f(x)-f(x^{\prime})|\leq\max(2,A)|x-x^{\prime}|^{\rho\alpha}(1+\max(|x|,|x^{\prime}|))^{\rho
M}.$

Proof of Lemma 1. Trivial. If
$|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{M}\geq 1$ then

$\displaystyle|f(x)-f(x^{\prime})|$ $\displaystyle\leq$ $\displaystyle 2$
$\displaystyle\leq$ $\displaystyle
2\left(|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{M}\right)^{\rho}$
$\displaystyle=$ $\displaystyle
2|x-x^{\prime}|^{\rho\alpha}(1+\max(|x|,|x^{\prime}|))^{\rho M}.$

If $|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{M}\leq 1$ then

$\displaystyle|f(x)-f(x^{\prime})|$ $\displaystyle\leq$ $\displaystyle
A|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{M}$ $\displaystyle\leq$
$\displaystyle
A\left(|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{M}\right)^{\rho}$
$\displaystyle=$ $\displaystyle
A|x-x^{\prime}|^{\rho\alpha}(1+\max(|x|,|x^{\prime}|))^{\rho M},$

proving the lemma.

_Remark_. Lemma 1 justifies the remark following the statement of Theorem 1.

Proof of Theorem 2. By Lemma 1 we can assume that

$|f(x)-f(x^{\prime})|\leq
D_{1}|x-x^{\prime}|^{\alpha}(1+\max(|x|,|x^{\prime}|))^{R/2}$

for all $x,x^{\prime}\in{{\bf R}^{d}}$. We assume $|x|\leq|x^{\prime}|$ and
consider two cases: i) $|x-x^{\prime}|^{\alpha}\geq(1+|x|)^{-R}$ and ii)
$|x-x^{\prime}|^{\alpha}\leq(1+|x|)^{-R}$. For case i) we have

$\displaystyle|f(x)-f(x^{\prime})|$ $\displaystyle\leq$
$\displaystyle|f(x)|+|f(x^{\prime})|$ $\displaystyle\leq$
$\displaystyle(1+|x|)^{-R}+(1+|x^{\prime}|)^{-R}$ $\displaystyle\leq$
$\displaystyle 2(1+|x|)^{-R}$ $\displaystyle\leq$ $\displaystyle
2(1+|x|)^{-R/2}$ $\displaystyle\leq$ $\displaystyle
2|x-x^{\prime}|^{\alpha/2}.$

For case ii) we have $|x-x^{\prime}|\leq 1$, so that $(1+|x^{\prime}|)\leq
2(1+|x|)$, implying

$\displaystyle|f(x)-f(x^{\prime})|$ $\displaystyle\leq$ $\displaystyle
D_{1}|x-x^{\prime}|^{\alpha}(1+|x^{\prime}|)^{R/2}$ $\displaystyle=$
$\displaystyle
D_{1}|x-x^{\prime}|^{\alpha/2}|x-x^{\prime}|^{\alpha/2}2^{R/2}(1+|x|)^{R/2}$
$\displaystyle\leq$ $\displaystyle D_{1}2^{R/2}|x-x^{\prime}|^{\alpha/2}.$

Conclusion: there are positive constants $\rho$ and $D_{3}$ such that, for all
$x,x^{\prime}\in{{\bf R}^{d}}$,

$|f(x)-f(x^{\prime})|\leq D_{3}|x-x^{\prime}|^{\rho}.$

Define

$\gamma:={\rho R^{\prime}\over R},$

which satisfies $0\leq\gamma<\rho$.

Take $|x|\leq|x^{\prime}|$ again and consider two cases: a)
$|x-x^{\prime}|^{\rho}\geq(1+|x|)^{-R}$ and b)
$|x-x^{\prime}|^{\rho}\leq(1+|x|)^{-R}$. For case a) we observe that

$\displaystyle|x-x^{\prime}|^{\rho-\gamma}$ $\displaystyle\geq$
$\displaystyle(1+|x|)^{-R(1-\gamma/\rho)}$ $\displaystyle=$
$\displaystyle(1+|x|)^{-R}(1+|x|)^{R\gamma/\rho}$ $\displaystyle=$
$\displaystyle(1+|x|)^{-R}(1+|x|)^{R^{\prime}},$

implying

$(1+|x|)^{-R}\leq|x-x^{\prime}|^{\rho-\gamma}(1+|x|)^{-R^{\prime}}.$

Since $(1+|x^{\prime}|)^{-R}\leq(1+|x|)^{-R}$, we also have
$|x-x^{\prime}|^{\rho}\geq(1+|x^{\prime}|)^{-R}$, and a similar argument
yields

$(1+|x^{\prime}|)^{-R}\leq|x-x^{\prime}|^{\rho-\gamma}(1+|x^{\prime}|)^{-R^{\prime}}\leq|x-x^{\prime}|^{\rho-\gamma}(1+|x|)^{-R^{\prime}}.$

Therefore, for case a),

$\displaystyle|f(x)-f(x^{\prime})|$ $\displaystyle\leq$
$\displaystyle|f(x)|+|f(x^{\prime})|$ $\displaystyle\leq$
$\displaystyle(1+|x|)^{-R}+(1+|x^{\prime}|)^{-R^{\prime}}$ $\displaystyle\leq$
$\displaystyle|x-x^{\prime}|^{\rho-\gamma}(1+|x|)^{-R^{\prime}}+|x-x^{\prime}|^{\rho-\gamma}(1+|x|)^{-R^{\prime}}$
$\displaystyle=$ $\displaystyle
2|x-x^{\prime}|^{\rho-\gamma}(1+|x|)^{-R^{\prime}}.$

For case b) we have:

$\displaystyle|f(x)-f(x^{\prime})|$ $\displaystyle\leq$ $\displaystyle
D_{3}|x-x^{\prime}|^{\rho}$ $\displaystyle=$ $\displaystyle
D_{3}|x-x^{\prime}|^{\rho-\gamma}|x-x^{\prime}|^{\gamma}$ $\displaystyle\leq$
$\displaystyle D_{3}|x-x^{\prime}|^{\rho-\gamma}(1+|x|)^{-R\gamma/\rho}$
$\displaystyle=$ $\displaystyle
D_{3}|x-x^{\prime}|^{\rho-\gamma}(1+|x|)^{-R^{\prime}},$

giving us (6) with $\beta:=\rho-\gamma$ and $D_{2}:=\max(2,D_{3})$. Theorem 2
is proved.

In wavelet/frame theory, (4) usually shows up with $R=d+\epsilon$. That makes
it natural to consider four possible sets of hypotheses on a function $f:{{\bf
R}^{d}}\to{\bf C}$.

$(I)$. There are positive constants $\epsilon$, $\alpha$, and a non-negative
$M$, such that:

For all $x\in{{\bf R}^{d}}$,

$|f(x)|\leq(1+|x|)^{-d-\epsilon}$

and, for all $x,x^{\prime}\in{{\bf R}^{d}}$,

$|f(x)-f(x^{\prime})|\leq|x-x^{\prime}|^{\alpha}\left(1+\max(|x|,|x^{\prime}|)\right)^{M}.$

$(II)$ There are positive constants $\epsilon$ and $\alpha$ such that:

For all $x\in{{\bf R}^{d}}$,

$|f(x)|\leq(1+|x|)^{-d-\epsilon}$

and, for all $x,x^{\prime}\in{{\bf R}^{d}}$,

$|f(x)-f(x^{\prime})|\leq|x-x^{\prime}|^{\alpha}.$

$(III)$ There are positive constants $\epsilon$ and $\alpha$ such that:

For all $x\in{{\bf R}^{d}}$,

$|f(x)|\leq(1+|x|)^{-d-\epsilon}$

and, for all $x,x^{\prime}\in{{\bf R}^{d}}$,

$|f(x)-f(x^{\prime})|\leq|x-x^{\prime}|^{\alpha}\left((1+|x|)^{-d-\epsilon}+(1+|x^{\prime}|)^{-d-\epsilon}\right).$

$(IV)$ There are positive constants $\epsilon$ and $\alpha$ such that:

For all $x\in{{\bf R}^{d}}$,

$|f(x)|\leq(1+|x|)^{-d-\epsilon}$

and, for all $x,x^{\prime}\in{{\bf R}^{d}}$,

$|f(x)-f(x^{\prime})|\leq|x-x^{\prime}|^{\alpha}\left((1+|x|)^{-d-\epsilon-\alpha}+(1+|x^{\prime}|)^{-d-\epsilon-\alpha}\right).$

It’s trivial that (neglecting unimportant multiplicative factors)
$(IV)\Rightarrow(III)\Rightarrow(II)\Rightarrow(I)$. Theorem 2 shows that,
with the same proviso, $(I)\Rightarrow(III)$. In [8, Definition 6.2, Lemma
6.2] it is shown that $(III)$ implies $(IV)$. Therefore $(I)$, $(II)$,
$(III)$, and $(IV)$ are “essentially equivalent” (equivalent modulo those
multiplicative factors). The implication $(I)\Rightarrow(IV)$ is probably the
most usesful of the lot. If $f$ satisfies $(IV)$ and has integral 0, we can
decompose it as a rapidly converging series of Hölder continuous, compactly
supported functions having cancelation, with good control on the functions’
sizes and supports [7, Lemma 3.5][8, Lemma 6.3]. Such decompositions have
value in their own right.

## References

  * [1] F. Abramovich, B. W. Silverman, “Wavelet Decomposition Approaches to Statistical Inverse Problems”, _Biometrika_ 85 (1998), 115–129.
  * [2] T. T. Cai, “On adaptive wavelet estimation of a derivative and other related linear inverse problems”, _Journal of Statistical Planning and Inference_ 108 (2002), 329–349.
  * [3] R. R. Coifman, Y. Meyer, _Wavelets: Calderón-Zygmund and multilinear operators_ , Cambridge University Press (1997), Cambridge.
  * [4] G. Didier, S. Jaffard, V. Pipiras, “On the vaguelet and Riesz properties of $L^{2}$-unbounded transformations of orthogonal wavelet bases”, _Journal of Approximation Theory_ 176 (2013), 94–117.
  * [5] R.-Q. Jia, “Bessel sequences in Sobolev spaces”, _Applied and Computational Harmonic Analysis_ 20 (2006), 298-311.
  * [6] H. Liu, H. Yang, Q. Yang, “Carleson Measures and Trace Theorem for $\beta$-harmonic Functions”, _Taiwanese Journal of Mathematics_ 22 (2018), 1107–1138.
  * [7] A. Uchiyama, “A constructive proof of the Fefferman-Stein decomposition of $BMO({\bf R}^{n})$”, _Acta Mathematica_ 148 (1982), 215–241.
  * [8] M. Wilson, _Weighted Littlewood-Paley Theory and Exponential-Square Integrability_ , Springer Lecture Notes in Mathematics 1924 (2007), New York.