positioning, shapes, arrows.meta, shadows, shapes.geometric, calc,
arrows.meta, shapes.misc

11institutetext: AWS AI Labs 22institutetext: Amazon AGI Foundations

# Diffusion Soup: Model Merging for Text-to-Image Diffusion Models

Benjamin Biggs∗ 11 Arjun Seshadri∗ 11 Yang Zou 22 Achin Jain 22 Aditya
Golatkar 11 Yusheng Xie 22 Alessandro Achille 11 Ashwin Swaminathan 22 Stefano
Soatto 11

###### Abstract

We present Diffusion Soup, a compartmentalization method for Text-to-Image
Generation that averages the weights of diffusion models trained on sharded
data. By construction, our approach enables training-free continual learning
and unlearning with no additional memory or inference costs, since models
corresponding to data shards can be added or removed by re-averaging. We show
that Diffusion Soup samples from a point in weight space that approximates the
geometric mean of the distributions of constituent datasets, which offers
anti-memorization guarantees and enables zero-shot style mixing. Empirically,
Diffusion Soup outperforms a paragon model trained on the union of all data
shards and achieves a 30% improvement in Image Reward (.34 $\to$ .44) on
domain sharded data, and a 59% improvement in IR (.37 $\to$ .59) on aesthetic
data. In both cases, souping also prevails in TIFA score (respectively, 85.5
$\to$ 86.5 and 85.6 $\to$ 86.8). We demonstrate robust unlearning—removing any
individual domain shard only lowers performance by 1% in IR (.45 $\to$
.44)—and validate our theoretical insights on anti-memorization using real
data. Finally, we showcase Diffusion Soup’s ability to blend the distinct
styles of models finetuned on different shards, resulting in the zero-shot
generation of hybrid styles.

**footnotetext: Equal contribution, alphabetical order. Correspondence to:
<EMAIL_ADDRESS>
##  1 Introduction

On paper, the ultimate goal of a foundational image generation model is to
approximate, given a large dataset $\mathcal{D}$, the underlying data
distribution $p(x)$ generating those images. This view, however, hides the
important subtleties that come with real-world data: any real large scale
training dataset $\mathcal{D}$ is not just an identically distributed
collection of images, but rather a variable entity where new data, coming from
different sources covering different domains and usage rights, is frequently
added or removed. In these situations, training a monolithic model on all data
is problematic. If the data changes, the whole model has to be retrained (at a
large cost) to either add new information (continual learning), or remove
information that cannot be used anymore (machine unlearning). Moreover, while
a single model trained on all the data together can have impressive coverage
and overall performance, it often underperforms expert models trained on
particular domains (see Table 1), to the detriment of downstream users
interested in those particular use-cases.

Figure 1: Diffusion Soup Enables Three Distinct Applications. (1) _Continual
Learning & Unlearning_: models trained on various data shards can be added to
improve performance or subtracted when removal is necessary. (2) _Zero-Shot
Style Mixing_ : souping blends the styles into a hybrid of its components with
no extra supervision. (3) _Anti-Memorization_ : Diffusion Soup prevents
memorization while capturing its high level style (note that we blur
depictions of inputs in this subfigure). Figure 2: Images from Diffusion Soup
Beat SD2.1 and a Combined Paragon. We visualize images generated by averaging
the weights of various models finetuned on data shards spanning different
categories (_Souped_), and compare them to images from the pretrained model
(_SD2.1_) and a paragon model trained all the shards (_Combined_). These
images highlight Diffusion Soup’s dominance in metrics (See Table 1) for Text-
Image Alignment, Aesthetics, and Fidelity. Best viewed in color and zoomed in.

This dilemma has prompted interest in compartmentalization or mixture of
expert (MoE) methods: rather than treating the model as a monolithic blackbox
trained on all data, different sets of parameters are trained on different
shards of the training set (corresponding, for example, to different domains
or data provided by different users) and then combined at inference time.
Compartmentalized models excel at continual learning, unlearning, and anti-
memorization tasks. However, in the simplest implementation through ensembling
of independent models, they significantly increase inference time — especially
as the number of models grows in the hundreds/thousands as would be desirable
to have fine control over data provenance. Moreover, ensemble models are
significantly more complex to deploy, as they require custom architectures and
proper load distribution across instances. The natural question is whether
there is a simple method to merge information between different models trained
on different subsets of data into a single model, with little or no additional
computational overhead.

To answer this question, we introduce Diffusion Soup. Diffusion Soup trains
separate models on different subsets of data, and then simply averages their
weights to obtain a single model. Averaging weights may seem ill-advised,
since weights do not live in a linear (vector) space and therefore averaged
weights may be far from, and perform worse than, any of its components.
However, we show that, when done properly, this strategy not only leads to
viable models, but actually outperforms a paragon monolithic model trained on
the combination of all data. See Fig. 2 for a visualization of results from
Diffusion Soup alongside a base model and the paragon.

Our result leverages the insight that the training process enforces linearity
by means of optimization with gradient descent: Each weight along the training
path is modified by an additive (linear) increment in a randomized direction.
Exploiting the structure of the update, we can show via Taylor expansion that
a Diffusion Soup of models finetuned from a shared pretrained checkpoint is
expected to (approximately) sample images from the geometric mean of the
individual model distributions. In contrast, training a monolithic model
samples from their arithmetic mean. Sampling from the geometric mean acts as
an implicit regularizer, which counters the decrease in quality due to the
overfitting commonly observed during fine-tuning of diffusion models. This
justifies our counter-intuitive results that a model soup performs better than
a monolithic expert trained with access to all the data.111While our results
are valid for local perturbations around a pretrained point, empirically we
show that the common pre-trained point can be generic enough that individually
trained experts starting from it achieve comparable performance than if they
were trained from scratch.

Empirically, we find that Diffusion Soup outperforms the comparable paragon
model trained on all the data shards in two settings: (1) when data shards are
grouped by domain (e.g. _Animals_ , _Electronics_ , etc.), Diffusion Soup
excels in TIFA score (85.5 $\to$ 86.5) and Image Reward (0.34 $\to$ 0.44); and
(2) when data shards are all geared towards Aesthetics, Diffusion Soup
prevails in TIFA score (85.6 $\to$ 86.8), Image Reward (0.37 $\to$ 0.59), and
CLIP Score (0.261 $\to$ 0.263).

We also show that Diffusion Soup: (1) can be used for training-free unlearning
(simply removing weights from the average); removing any domain’s shard
decreases performance by at most 1% in Image Reward (.45 $\to$ .44); (2) the
souped model approximates a Near Access Freeness condition, which provides the
souped model with better anti-memorization guarantees; (3) can be modified by
the user to achieve desired zero-shot blending between styles.

##  2 Related Work

Diffusion models. Diffusion models [17, 10, 36, 39, 34] are state-of-the-art
models used for text-to-image generation. These models add Gaussian noise to
an image (or its latents) in the forward process, and learn the score function
to denoise the image in the backward process (introduced by [25]) during
generation. [35, 34] condition diffusion models on text to enable text-to-
image generation at inference using a technique called classifier-free
guidance[18]. [41, 40] show that the forward-backward process obeys a
stochastic differential equation, we use this framework in this paper.
Compositional generation. [26, 12, 15, 3, 5, 51] propose performing
compositional generation by merging output flows of diffusion models, methods
that often incur increased compute cost. [48] proposes a mixture of expert
method by replacing the standard convolutional layers with mixture-of-expert
layers which reduce the inference cost and the number of parameters during
inference. We show that our method further reduces the inference cost and
parameters–and in fact incurs no additional inference or memory overhead
beyond that of a single model. Model merging. Weight averaging [13, 21, 7, 30,
46] is a popular technique used to improve the accuracy of discriminative
models. [28, 27] further extended these methods for linearized convolutional
and transformer based models. [2] proposed a model averaging method for
generative adversarial networks, however, such methods have been unexplored
for diffusion models. To the best of our knowledge, we are the first method to
show that weight averaging not only generates high quality images, but also
improves the text-to-image alignment and reduces memorization at inference.

##  3 Preliminaries

Latent diffusion models like Stable Diffusion [35] aim to model a data
distribution $p(x_{0})$ of images $x_{0}\in\mathcal{X}$ which can be sampled
at inference time. We use the stochastic differential equations (SDE) based
diffusion formalism of [41, 22] to define the basics of diffusion models.
Given the initial distribution $p(x_{0})$ at time $t=0$, SDE based formalism
transforms it into a reference distribution $p(x_{1})=\mathcal{N}(0,I)$
(Gaussian distribution) at time $t=1$ in the forward process:

$dx_{t}=-\dfrac{1}{2}\beta_{t}x_{t}dt+\sqrt{\beta_{t}}d\rho_{t}$ (1)

where $x_{t}$ is the diffused latents at $t$, $d\rho_{t}$ is the standard
Wiener process, and $\beta_{t}$ are time varying diffusion coefficients used
to manipulate the signal-to-noise ratio of the diffusion process. The
intermediate latents $x_{t}$ are distributed according to a conditional
gaussian distribution
$p(x_{t}|x_{0})=\mathcal{N}(x_{t};\gamma_{t}x_{0},\sigma_{t}^{2}I)$ by
construction where $\gamma_{t}=\exp(-\dfrac{1}{2}\int_{0}^{t}\beta_{t}dt)$ and
$\sigma_{t}^{2}=1-\gamma_{t}^{2}$, providing the following marginal:
$p(x_{t})=\int_{x_{0}}p(x_{t}|x_{0})dx_{0}$. [25, 41] shows that the forward
process in Eq. 1 can be effectively reversed by the backward diffusion process
given by:

$dx_{t}=-\dfrac{1}{2}\beta_{t}x_{t}dt+\sqrt{\beta_{t}}d\rho_{t}-\nabla_{x_{t}}\log
p(x_{t})dt$ (2)

where $dt$ is a decrement in time corresponding to the backward process,
introduced by [25]. Eq. 2 reduces sampling from $p(x_{0})$ to iterative
application of $\nabla_{x_{t}}\log p(x_{t})$ (score function) given an initial
noisy samples $x_{1}\sim\mathcal{N}(0,I)$. The score function
$\nabla_{x_{t}}\log p(x_{t})$ is generally difficult to estimate in practice
and is instead learned using a neural network $\epsilon_{w}(x_{t},t)$ and a
training dataset $D$ through score-matching [20, 40, 11, 41]. To enhance the
usability and control of diffusion models they are often conditioned with
textual prompts $y$ to model the conditional distribution $p(x_{0}|y)$ which
modifies the score-estimating neural network as $\epsilon_{w}(x_{t},t,y)$, to
now accept $y$ as input during training and inference. Incorporating all these
subtleties results in the following optimization problem to train text-to-
image diffusion models:

$\text{min}_{w}\mathbb{E}_{(x_{0},y)\sim
p(x_{0},y)}\mathbb{E}_{t}[\|\epsilon_{w}(x_{t},t,y)+\nabla_{x_{t}}\log
p(x_{t}|x_{0})\|]$ (3)

We can obtain an equivalent optimization problem by formulating the forward
process in Eq. 1 as a Markov chain and optimizing a variant of the evidence
lower bound (ELBO) [17, 39].

##  4 Diffusion Soup

While Diffusion models are pre-trained on large monolithic datasets (e.g.,
LAION-5B), in downstream applications it is often more common to have datasets
$D_{i}$ come from different data sources, which may correspond to different
data providers, or different domains. We use $\\{p^{(i)}(x_{0})\\}_{i}$ to
represent the collection of distributions corresponding to datasets,
$\\{D_{i}\\}_{i=1}$. Ensembling the outputs of generative models [15, 3]
integrates information from different data sources, but is expensive due to
the size of the models. Instead we propose to ensemble the weights of the
model inspired by recent work [23, 1, 29, 14, 45, 50, 28] showing that the
fine-tuning dynamics of the large models can be approximated with a Taylor
series approximation:

$\epsilon_{w_{c}+\Delta
w_{i}}(x_{t},t,y)\approx\epsilon_{w_{c}}(x_{t},t,y)+\nabla_{w}\epsilon_{w_{c}}(x_{t},t,y)|_{w=w_{c}}\Delta
w_{i}$ (4)

where $\Delta w_{i}$ is the perturbation of the weights learned during fine-
tuning. This result can be leveraged to show that ensembling the outputs
corresponds to ensembling the weights of the models $w_{i}$ trained on
different data sources $D_{i}$ (sampled from $p^{(i)}(x_{0})$). More
precisely, we define the souped prediction as:

$\displaystyle\epsilon_{\text{soup}}$
$\displaystyle\triangleq\underbrace{\epsilon_{\sum_{i}k_{i}w_{i}}}_{\text{Souping}}(x_{t},t,y)=\epsilon_{\sum_{i}k_{i}(w_{c}+\Delta
w_{i})}(x_{t},t,y)$
$\displaystyle\stackrel{{\scriptstyle(a)}}{{\approx}}\epsilon_{w_{c}}(x_{t},t,y)+\nabla_{w}\epsilon_{w_{c}}(x_{t},t,y)|_{w=w_{c}}\cdot\Big{(}\sum_{i}k_{i}\Delta
w_{i}\Big{)}$
$\displaystyle=\sum_{i}k_{i}\Big{(}\epsilon_{w_{c}}(x_{t},t,y)+\nabla_{w}\epsilon_{w_{c}}(x_{t},t,y)|_{w=w_{c}}\Delta
w_{i}\Big{)}$
$\displaystyle\stackrel{{\scriptstyle(b)}}{{\approx}}\sum_{i}k_{i}\epsilon_{w_{c}+\Delta
w_{i}}=\sum_{i}k_{i}\epsilon_{w_{i}}(x_{t},t,y)=\epsilon_{\text{ensemble}}$
(5)

where $k_{i}>0$, such that $\sum_{i}k_{i}=1$ is a hyper-parameter which can be
tuned by the user, and $(a),(b)$ follow from using the first order Taylor
series approximation of the network. To summarize, Diffusion Soup approximates
ensembling, and involves fine-tuning $n-$diffusion models
($\\{\epsilon_{w_{i}}(x_{t},t,y)\\}_{i}$) on $n-$data sources
($\\{p^{(i)}(x_{0})\\}_{i}$) respectively, and averaging the parameters.

###  4.1 Sampling Distribution for Souping

Modifying the score (or the $\epsilon(x_{t})$) during the backward diffusion
process changes the distribution which is sampled by the model. For instance,
using Eq. 2 samples images from the distribution $p(x_{0})$. In our case,
since we modify the individual $\\{\epsilon_{w_{i}}(x_{t},t,y)\\}$ to the soup
$\epsilon_{\sum_{i}k_{i}w_{i}}(x_{t},t,y)$ it is essential to identify the
sampling distribution of the model. Depending on the choice of $k_{i}$ we can
show that the souped model can either sample from various aggregations of the
set of distributions $\\{p^{(i)}(x_{0})\\}$. This is described in the
following results:

###### Proposition 1

(Geometric Mean) Let $\nabla_{x_{t}}\log p^{(i)}(x_{t})$ be the marginal score
in Eq. 2 where $x_{t}=\gamma_{t}x_{0}+\sigma_{t}\epsilon$, with $x_{0}\sim
p^{(i)}(x_{0})$ and $\epsilon\sim\mathcal{N}(0,I)$. Let
$\epsilon_{w_{i}}(x_{t},t,y)$ be a neural network with sufficient capacity
trained to match $\nabla_{x_{t}}\log p^{(i)}(x_{t})$. Under Eq. 2’s conditions
on the sampling procedure, $(1/n)\sum_{i=1}^{n}\nabla_{x_{t}}\log
p^{(i)}(x_{t})$ generates samples from
$Z^{-1}\big{(}\prod_{i}p^{(i)}(x_{0})\big{)}^{1/n}$. Furthermore, using Eq. 4,
$\epsilon_{\text{soup}}$ with $k_{i}=1/n$ generates samples from
$Z^{-1}\big{(}\prod_{i}p^{(i)}(x_{0})\big{)}^{1/n}$.

The previous result states Diffusion Soup with $k_{i}=1/n$ generates samples
from the geometric mean of the distribution of the individual models, which
can be useful to provide memorization-free generation [44]. Conversely, the
following proposition shows souping (with an appropriately chosen value of
$k_{i}$) samples from the union of all data (equivalent to training a model on
the union).

###### Proposition 2

(Arithmetic Mean) Let $\nabla_{x_{t}}\log p^{(i)}(x_{t})$ be the marginal
score in Eq. 2 where $x_{t}=\gamma_{t}x_{0}+\sigma_{t}\epsilon$, with
$x_{0}\sim p^{(i)}(x_{0})$ and $\epsilon\sim\mathcal{N}(0,I)$. Let
$\epsilon_{w_{i}}(x_{t},t,y)$ be a neural network (with sufficient capacity)
trained to match $\nabla_{x_{t}}\log p^{(i)}(x_{t})$. Sampling using
$(1/n)\sum_{i=1}^{n}\lambda_{i}\frac{p^{(i)}(x_{t})}{p(x_{t})}\nabla_{x_{t}}\log
p^{(i)}(x_{t})$ with Eq. 2 generates samples from
$\sum_{i}\lambda_{i}p^{(i)}(x_{0})$, where
$p(x_{t})=\sum_{i}\lambda_{i}p^{(i)}(x_{t})$, $\sum_{i}\lambda_{i}=1$.
Furthermore, using Eq. 4, $\epsilon_{\text{soup}}$ with
$k_{i}=\lambda_{i}\frac{p^{(i)}(x_{t})}{p(x_{t})}$ generates samples from
$\sum_{i}\lambda_{i}p^{(i)}(x_{0})$.

Depending on the choice of $k_{i}$ in souping, we show that our model can
sample from different distributions. Note that in Proposition 2, $k_{i}$
depends on the $t$ and thus requires re-souping per prompt at every timestep.
In this work, $k_{i}$ is fixed across prompts and timesteps, and set to be
uniform ($k_{i}=1/n$) or chosen using various greedy approaches, as we detail
below.

###  4.2 Greedy Souping

In Eq. 5 we show that the optimal weights after souping are a linear
combination of the individual weights $w_{i}$ (data sources/experts),
$w_{\text{soup}}=\sum_{i}k_{i}w_{i}$, where we treat $k_{i}$ as hyper-
parameters to be optimized before inference subject to the constraint
$\sum_{i}k_{i}=1$. Let $L(w_{\text{soup}},D_{\text{val}})$ be a evaluation
metric used assess the quality of the diffusion model, for example, TIFA score
or Image Reward. The following optimization problem solves for the optimal
coefficients $k_{i}$:

$\\{k_{i}^{*}\\}_{i}=\operatornamewithlimits{argmin}_{\sum_{i}k_{i}=1,k_{i}>0}\textstyle
L\big{(}\sum_{i}k_{i}w_{i},D_{\text{val}}\big{)}$ (6)

Obtaining a closed form expression for Eq. 6 is intractable for diffusion
model evaluation, and so we consider two greedy approaches to obtain
coefficients. The first, Greedy Soup begins with the best performing
individual weight and incrementally soups weights in order of individual
performance—keeping the weights if they improve performance and discarding
them if they do not. The second, Reverse Greedy Soup begins with the uniform
soup and removes weights in order of increasing performance.

##  5 Analysis

###  5.1 Near Access Freeness of Diffusion Soup

Diffusion models often memorize some of their training data [4, 38] posing a
risk of reproducing samples from the training dataset. [44] recently proposed
a mathematical framework based on near-access freeness (NAF) to prevent
generative models from memorizing samples present in the training data. Let
$D$ be the dataset with some samples $C=\\{c_{i}\\}_{i}$, and $D=D_{1}\bigcup
D_{2}$ be two disjoint splits of the dataset. Let $p^{(1)}(x|y),p^{(2)}(x|y)$
(where $x$ is the image and $y$ is the caption) be two diffusion models
trained on $D_{1},D_{2}$ respectively. Then Algorithm 3 from [44]
(CP$-\Delta$) shows that sampling from the geometric mean
$p(x|y)=Z^{-1}\sqrt{p^{(1)}(x|y)p^{(2)}(x|y)}$ provides anti-memorization
guarantees with respect to any sample $c_{i}$ present in only one of $D_{1}$
or $D_{2}$. That is, it satisfies $\varepsilon$-NAF:

$\Delta(p(x|y)||\text{safe}_{C}(x|y))\leq\varepsilon$ (7)

where $\text{safe}_{C}(x|y)$ is a model which has not been trained on $c_{i}$,
$\Delta$ is a divergence between two distributions (e.g., Kullback-Leibler
divergence). Sampling from $p(x|y)=Z^{-1}\sqrt{p^{(1)}(x|y)p^{(2)}(x|y)}$ is
difficult for diffusion models. However, sampling can be performed implicitly
in the backward step during score computation [15, 12] as $\nabla\log
p(x|y)=(1/2)\big{(}\nabla\log p^{(1)}(x|y)+\nabla\log p^{(2)}(x|y)\big{)}$.
With this observation we have the following result:

###### Proposition 3

($\varepsilon$-NAF for Diffusion Soup) Let
$\epsilon_{\text{soup}}(x_{t},t,y)\triangleq\epsilon_{\frac{1}{2}(w_{1}+w_{2})}(x_{t},t,y)$
be the soup model obtained by training $w_{i}$ on $D_{i}$, such that
$D=D_{1}\cup D_{2}$, and $D_{1}\cap D_{2}=\varnothing$. Then, under certain
conditions, sampling using $\epsilon_{\text{soup}}(x_{t},t,y)$ satisfies
$\varepsilon$-NAF for some $\varepsilon$ which depends only on $D_{1}$ and
$D_{2}$.

The above proposition suggests that, for different choices of $D_{1}$ and
$D_{2}$ we can guarantee different subsets $C=\\{c_{i}\\}\subset D$ of NAF-
protected samples. If all training samples require anti-memorization ($C=D$),
then it suffices to pick $D_{1}$ and $D_{2}$ to be disjoint partitions of $D$.
Conversely, in a mixed privacy setting [16], we have a public set
$D_{\text{publ}}$ set for which anti-memorization is not required, and a
private set $D_{\text{priv}}$ that requires anti-memorization. In this case,
we can make better use of the data by picking $D_{1}=D_{\text{publ}}$ to be a
core safe set used to pre-train the model, and $D_{2}=D_{\text{publ}}\cup
D_{\text{priv}}$ to be the fine-tuning set. We explore both ideas for
Diffusion models in Section 6.2.

###  5.2 Computational efficiency

Let $M$ denote the memory consumption (i.e. parameters) during inference, $T$
the time complexity of a forward pass. Compartmentalizing $n$ models has a
worst case complexity of $\mathcal{O}(nM),\mathcal{O}(nT)$ for the memory and
time respectively, while ensembling [5, 51, 48] methods have
$\mathcal{O}(nM),\mathcal{O}(T)$ complexity respectively. Diffusion Soup
reduces the memory complexity by averaging the weights into a single model,
resulting in an optimal $\mathcal{O}(M),\mathcal{O}(T)$ complexity. Souping
reaps the benefits of multiple specialists and data sources without additional
overhead.

###  5.3 Unlearning

Given $n$ different data sources $D_{f}=\\{D_{i}\\}_{i}$, and their
corresponding diffusion model weights $w_{i}$, Diffusion Soup provides a
single set of weights $w_{\text{soup}}=\sum_{i}k_{i}w_{i}$. Just as adding new
data sources to the soup is straightforward, removal is just as easy: should a
data source $D_{i}$ decides to withdraw their data, we can simply update the
souped weights as,
$w_{\text{soup}}:=\frac{w_{\text{soup}}-k_{i}w_{i}}{1-k_{i}}$. Should only a
subset of data from $D_{i}$ be removed, we can simply reset the diffusion
model $\epsilon_{w_{i}}$ to weights $w_{c}$, and fine-tune it with the
remaining data in $D_{i}$. Souping thus enables efficient disgorgement of
data.

Table 1: Souping a collection of specialist models produces a high-quality
generalist. We consider data shards that allow the model to specialize in a
range of image domains. Souping these specialist together outperforms all
individual models. See Figure 1 for a visualization of Diffusion Soup’s
performance.

|  | TIFA | IR | CLIP
---|---|---|---|---
Base |  SD2.1 |  85.5 |  0.30 |  0.259
SFT |  Animals (AN) |  86.0 |  0.41 |  0.256
Body Parts (BP) |  85.6 |  0.28 |  0.257
Electronics (EL) |  85.9 |  0.42 |  0.260
Accessories (AC) |  85.3 |  0.38 |  0.257
Clothes (CL) |  85.0 |  0.35 |  0.256
Food (FO) |  85.7 |  0.37 |  0.259
Hardlines (HA) |  85.6 |  0.35 |  0.260
Products (PR) |  85.3 |  0.37 |  0.258
Vehicles (VE) |  85.2 |  0.33 |  0.257
Combined |  All |  85.5 |  0.34 |  0.257
Souped |  Uniform |  86.2 |  0.45 |  0.260
Reverse Greedy |  86.3 |  0.43 |  0.260
Greedy |  86.5 |  0.44 |  0.259

##  6 Experiments

We empirically evaluate Diffusion Soup’s ability to achieve training free
continual learning and unlearning, provide anti-memorization guarantees, and
blend disparate styles into unique hybrids. Our experiments fall into five
broad application domains: (1) specialist aggregation, (2) aesthetic
enhancement, (3) unlearning (4) blending artistic styles, and (5) anti-
memorization. We use Stable Diffusion 2.1 (SD2.1) as the pretrained model, and
follow the finetuning procedure proposed in [9]. Our results are not unique to
SD2.1, and broadly apply to any Diffusion Model. Details about hyperparameters
are in the supplement.

###  6.1 Datasets and Evaluation Metrics

####  6.1.1 Datasets

We use publicly-available datasets for our experiments. (1) Souping
Specialists: We generate category-specific datasets and finetune on them to
obtain category specialists. These datasets are constructed by first using
CLIP [33] to retrieve the top 10K highest scoring images for each category,
and then using the LAION aesthetic score [37] predictor to retain images with
an aesthetic score above 6, leaving approximately a thousand images per
category. (3) Style Mixing: We use the Pokemon dataset [32] which contains
images of Pokemon characters along with BLIP [24] captions, and the FS-COCO
dataset [8] to generate finetuned checkpoints. (4) Anti-Memorization and
Unlearning: We use the Pokemon dataset to evaluate anti-memorization and
MSCOCO for unlearning.

Figure 3: Finetuning Models on Category Specific Data Subsets Produces
Specialists. We visualize results from finetuning SD2.1 models on various data
subsets shards by category. The finetuning process specializes the diffusion
model to the subset’s category: for example, the Body Parts dataset enhances
the prominence of fingers, and the Fashion Clothes dataset enhances outfits.
These models can be souped together to obtain generalized models that
outperform all specialists (See Table 1).

####  6.1.2 Evaluation Metrics

We consider three metrics commonly used in benchmarking the performance of
diffusion models: (1) Text-to-Image Faithfulness evaluation with Question
Answering[19] (TIFA Score) measures the faithfulness of a generated image to
its text input via visual question answering (VQA), which we apply to 2k
collected prompts from the MSCOCO Dataset. (2) Image Reward [47] (IR) is a
scoring model that captures human preferences on image-text alignment,
fidelity, and harmlessness. (3) CLIP Score [33] (CLIP) captures text-image
alignment via the cosine similarity between CLIP embeddings for the generated
image and the text prompt.

###  6.2 Applications

####  6.2.1 Continual Learning from Multiple Data Subsets

The different data subsets that compartmentalization methods operate on are
not chosen in practice but rather imposed by the changing nature of data. We
therefore showcase the performance of Diffusion Soup on two broad groupings
that data subsets often fall into: different domains of data with different
usage rights, or task-specific data provided by different users.

##### Soup of Specialists.

To represent different domains of data, we construct shards corresponding to
diverse categories such as _Animals_ , _Food_ , _Fashion Clothes_ , etc and
finetune SD2.1 to obtain domain specialists corresponding to each shard. Fig.
3 visualizes each specialist’s ability to improve features corresponding to
their domain over SD2.1, e.g. _Animals_ improve animal photorealism. We now
compare various souping methods alongside these specialists and the paragon
model trained on the union of all shards, and showcase our findings in Table
1. We find that our finetuned specialists both outperform base SD2.1 in TIFA
and IR, and often outperform the combined model paragon, confirming prior work
[15, 3] demonstrating the value of training specialist models. A uniform soup
of the specialist models outperforms all specialists and the paragon in TIFA,
IR, and CLIP Score, indicating superior image quality and alignment compared
to all base and individual specialist models. Our two greedy souping
approaches further optimize TIFA and achieve the highest performing TIFA
scores of 86.3 and 86.5. These image alignment optimizations come at a slight
cost in image quality however, as indicated by the slight decrease in IR.
Regardless, the approaches outperform all specialists and the combined model
paragon in TIFA and IR, resulting in a Pareto optimal curve of souping options
to choose from.

Table 2: Diffusion Soup Outperforms Base and Combined Models in Aesthetic
Enhancement. We consider data shards that all have one task in common,
aesthetic enhancement. Our souped results outperform all base models and the
combined model in which a single model is trained on a union of the datasets.

| Base & SFT | Combined (Cumulative) | Souped (Cumulative)
---|---|---|---
|  TIFA |  IR |  CLIP |  TIFA |  IR |  CLIP |  TIFA |  IR |  CLIP
SD2.1 |  85.5 |  0.30 |  0.259 |  - |  - |  - |  - |  - |  -
AE1 |  86.5 |  0.53 |  0.263 |  86.5 |  0.53 |  0.263 |  86.5 |  0.53 |  0.263
AE2 |  86.3 |  0.54 |  0.261 |  86.0 |  0.40 |  0.261 |  86.7 |  0.60 |  0.263
AE3 |  86.1 |  0.47 |  0.262 |  85.7 |  0.38 |  0.261 |  86.8 |  0.60 |  0.263
AE4 |  86.2 |  0.43 |  0.260 |  86.0 |  0.39 |  0.261 |  86.8 |  0.59 |  0.263
AE5 |  85.9 |  0.45 |  0.262 |  85.6 |  0.37 |  0.261 |  86.8 |  0.59 |  0.263

We visualize the improvements of Diffusion Soup over the base model and the
combined model paragon in Fig. 1, highlighting better image alignment, fewer
artifacts, and improved aesthetics. Unlike other approaches such as ensembling
or MoE methods, these improvements come without compromising on memory
consumption and runtime, a significant advance for compartmentalization.

##### Enhancing Image Aesthetics.

We next consider data shards all pertaining to a single task, aesthetic
enhancement. We construct 5 highly aesthetic subsets of MSCOCO, AE1-5, ordered
by decreasing aesthetic quality to simulate the diversity among data shards
stemming from different providers. Once again, we compare various souping
approaches to models trained on each shard, as well as a paragon trained on
the union. To shed further light into the gap between the Diffusion Soup and
the Paragon, we show cumulative results for both as we take each aesthetic
subset into consideration.

Our results are highlighted in Table 2. We find that all finetuned models
outperform base SD2.1 in TIFA and IR, but especially on IR, indicative of the
aesthetic quality improvements that IR is particularly sensitive to. Once
again, we find that a uniform Diffusion Soup outperforms base SD2.1, all
finetuned models, and the combined paragon model on TIFA, IR and CLIP,
respectively obtaining scores of 86.8, .59, and .263. The dominance of
Diffusion Soup is also shown cumulatively, with the method outperforming the
combined paragon model with every additional aesthetic dataset. As continual
learning progresses over a growing number of dataset shards, Diffusion Soup
continues to improve in performance and increase its lead on a combined model
paragon. Curiously, we find that some individual models outperform the
combined model paragon – while these models are not specialists in the sense
of the previous section, they appear to still specialize in the particular
types of aesthetic enhancements present in each data shard, some of which
better optimize for TIFA and IR than a combined dataset.

####  6.2.2 Machine Unlearning

We next consider settings where data must be removed, or cannot be used
anymore. For Diffusion Soup, just as adding data shards consisted simply of
averaging models trained on those shards, removing shards can be trivially
implemented through weighted subtraction. We demonstrate unlearning using our
category datasets from which domain specialists are constructed, demonstrating
that domain specialists can be removed from our model soup without
meaningfully compromising the performance of the soup. Remarkably, we find
that removing any specialist reduces the performance of the soup by at most 1%
in IR. Fig. 4 displays our full results, and shows that leaving any one shard
out still produces a model soup that closely mirrors the performance of
uniform soup in all metrics, and continues to outperform SD2.1 even in the
worst case scenario, the removal of the _Clothes_ model. In some cases the
removal of models improves the TIFA score, and this is the principle exploited
by the Reverse Greedy Soup. However, in practice one cannot select the data
that is removed, and we view our Fig. 4 as supporting a uniform performance
guarantee. The cumulative souping results of Table 2 when read from bottom up
serves as yet another demonstration of robust unlearning; indeed a model soup
removing data shards AE4 and AE5 performs identically in TIFA score to the
uniform soup over all shards, and further removal only results in a mild
reduction in performance.

Figure 4: Removing Any Individual Data Shard Does Not Meaningfully Reduce
Performance. We soup specialists leaving one out at a time to demonstrate that
no individual specialist significantly affects the quality of the generalist.
Our results show that Diffusion Soup can be used for Machine Unlearning. From
left-to-right, graphs show TIFA, Image Reward and CLIP Score. Performance of
uniform soup model is in _green_ , and SD2.1 in _red_.

####  6.2.3 Zero-Shot Style Mixing

We employ Diffusion Soup to merge the distinct styles of two separate
datasets— FS-COCO’s sketch-like imagery and Pokemon illustrations—in a zero-
shot manner, i.e. without the need for hybrid training samples. Fig. 5
displays the results of this style fusion. The first row highlights the model
tuned on the Pokemon dataset and the second row showcases the FS-COCO-tuned
model which generates monochrome line-art sketches. The third row demonstrates
that that souping these two models yields generation of a new hybrid style of
line-art cartoon characters, without using any hybrid training examples. Our
findings are the first to link model souping to Style Mixing, and stem from
our novel application of souping to Diffusion Models, and more broadly
generative models. We hypothesize that Style Mixing derives from Diffusion
Soup sampling from the Geometric Mean of the constituent distributions,
formalized in Proposition 1, and therefore distinguishes our results from the
outputs produced by training a model on the union of the two datasets. We
leave further exploration of this hypothesis for future work.

Figure 5: Diffusion Soup merges finetuned models to create hybrid styles. We
apply Diffusion Soup to models finetuned on Pokemon (_Row 1_) and FS-COCO
(_Row 2_) to create a hybrid style (_Row 3_). The results are zero-shot since
we do not have examples of the hybrid style for training. Figure 6: Diffusion
Soup reduces dataset memorization. (Bottom) Models (e.g. Pokemon-Full) trained
on Pokemon data (Original) tend to reproduce some training samples at
inference time. Note that we blur depictions of inputs in this subfigure.
However, souping models (Pokemon-Soup) trained on disjoint subsets of Pokemon
(S1 and S2) significantly reduces the memorization when compared to the
Pokemon-Full. The same holds for cross-domain souping (Pokemon-_Animals_
-Soup), which soups the Pokemon model (Pokemon-Full) with a model trained on a
safe dataset (_Animals_ -Full). (Top) We formalize this anecdotal result with
a plot of the CLIP distance of various datasets to the Original Pokemon
dataset (_Top Left_). Larger distance or mass towards right implies lower
memorization. Souping Pokemon subsets increases the CLIP distance (top middle)
and therefore reduces reproduction of Pokemon. Souping Pokemon with _Animals_
reduces memorization further (_Top Right_). _Prompt used in figure: “a cartoon
character with a star on his head”_.

####  6.2.4 Anti-Memorization

We demonstrate that souping reduces memorization in Diffusion Models with two
distinct scenarios, and display our results in Fig. 6. In the first scenario,
we randomly split the Pokemon dataset into two shards S1 and S2, and train
separate models on each shard. Even though both models reproduce some Pokemon,
their soup surprisingly does not. We visualize an example in the bottom-middle
of Fig. 6, where Pokemon-Soup avoids reproducing samples from the Pokemon
dataset while capturing its style. We formalize this finding at a distribution
level using CLIP scores. Taking each prompt from the Original Pokemon Dataset,
we measure the distance between the Original images and model generated images
using CLIP scores. These values together chart a distance distribution of
model outputs to the original dataset, and we visualize such distributions in
the top-left of Fig. 6. There, the two models trained on shards (shown in
_blue_ and _orange_) each reproduce some samples from the original dataset, as
evidenced by low CLIP score peaks of the distributions. The souped model’s
distance distribution (in _green_) however is shifted considerably to the
right, demonstrating that it generates images which are far from the Original
data, i.e. avoids memorizing the dataset.

In the second scenario, we soup a model trained on the complete Pokemon
dataset with a model trained on a completely disjoint domain—the _Animals_
model from Table 1. We visualize an example in the bottom-left of Fig. 6,
where Pokemon-_Animals_ -Soup once again avoids memorization while capturing
style. We plot distance distributions in the top-middle of Fig. 6, where the
Pokemon model (_blue_) reproduces the Pokemon data, and the _Animals_ model
(_orange_) is unsurprisingly far away from it. The souped model’s distribution
(_green_) again sits to the right of that of the Pokemon model, indicative of
its reduced memorization. To compare the effect of the two souping approaches,
we compare Pokemon-Soup and Pokemon-_Animals_ Soup’s distance distributions in
the top right of Fig. 6, and find that the latter reduce memorization more
than the former, but requires an additional dataset and corresponding model.
These two methods are those both effective anti-memorization strategies, each
with unique tradeoffs for a practitioner to navigate.

####  6.2.5 Limitations

Diffusion Soup performs a variety of tasks with no additional inference cost,
yet it still requires training $n$ model shards, and continual training on
every new data shard. Also, averaging weights with a particular weighting of
each model must be performed globally, and cannot be performed per sample
without incurring the inference costs associated with ensembling. Although
Adapters can help reduce both the training and reweighting burden, they come
at the cost of reduced performance.

##  7 Conclusion

We present Diffusion Soup, a novel compartmentalization method in the image
generation domain. Our method aggregates the benefits of many Text-to-Image
models without the costly memory and inference time overhead of ensembling.
Diffusion Soup outperforms constituent models, and a combined model paragon in
specialist aggregation and aesthetic enhancement. Furthermore, Diffusion Soup
enables two applications: the mixing of disparate artistic styles into novel
hybrid styles and the ability to avoid memorizing images while capturing the
underlying informational content. While our contribution reflects the first
introduction of model souping to large scale generative AI, its potential to
impact performance in other domains is untold. We leave as future work to
explore souping in other domains such as language modeling, or visual question
answering — domains that can greatly magnify the impact of our work.

## References

  * [1] Achille, A., Golatkar, A., Ravichandran, A., Polito, M., Soatto, S.: Lqf: Linear quadratic fine-tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15729–15739 (2021)
  * [2] Avrahami, O., Lischinski, D., Fried, O.: Gan cocktail: mixing gans without dataset access. In: European Conference on Computer Vision. pp. 205–221. Springer (2022)
  * [3] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al.: ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)
  * [4] Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., Wallace, E.: Extracting training data from diffusion models. In: 32nd USENIX Security Symposium (USENIX Security 23). pp. 5253–5270 (2023)
  * [5] Chen, Z., Deng, Y., Wu, Y., Gu, Q., Li, Y.: Towards understanding mixture of experts in deep learning. arXiv preprint arXiv:2208.02813 (2022)
  * [6] Cheng, X., Bartlett, P.: Convergence of langevin mcmc in kl-divergence. In: Algorithmic Learning Theory. pp. 186–211. PMLR (2018)
  * [7] Choshen, L., Venezian, E., Slonim, N., Katz, Y.: Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.03044 (2022)
  * [8] Chowdhury, P.N., Sain, A., Bhunia, A.K., Xiang, T., Gryaditskaya, Y., Song, Y.Z.: Fs-coco: Towards understanding of freehand sketches of common objects in context. In: ECCV (2022)
  * [9] Dai, X., Hou, J., Ma, C.Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., Yu, M., Kadian, A., Radenovic, F., Mahajan, D., Li, K., Zhao, Y., Petrovic, V., Singh, M.K., Motwani, S., Wen, Y., Song, Y., Sumbaly, R., Ramanathan, V., He, Z., Vajda, P., Parikh, D.: Emu: Enhancing image generation models using photogenic needles in a haystack (2023)
  * [10] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34, 8780–8794 (2021)
  * [11] Dockhorn, T., Vahdat, A., Kreis, K.: Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:2112.07068 (2021)
  * [12] Du, Y., Durkan, C., Strudel, R., Tenenbaum, J.B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., Grathwohl, W.S.: Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. In: International conference on machine learning. pp. 8489–8510. PMLR (2023)
  * [13] Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D.P., Wilson, A.G.: Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems 31 (2018)
  * [14] Golatkar, A., Achille, A., Ravichandran, A., Polito, M., Soatto, S.: Mixed-privacy forgetting in deep networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 792–801 (2021)
  * [15] Golatkar, A., Achille, A., Swaminathan, A., Soatto, S.: Training data protection with compositional diffusion models. arXiv preprint arXiv:2308.01937 (2023)
  * [16] Golatkar, A., Achille, A., Wang, Y.X., Roth, A., Kearns, M., Soatto, S.: Mixed differential privacy in computer vision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8376–8386 (2022)
  * [17] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [18] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
  * [19] Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., Smith, N.A.: Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897 (2023)
  * [20] Hyvärinen, A., Dayan, P.: Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research 6(4) (2005)
  * [21] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 (2018)
  * [22] Karras, T., Aittala, M., Aila, T., Laine, S.: Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems 35, 26565–26577 (2022)
  * [23] Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., Pennington, J.: Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems 32 (2019)
  * [24] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: ICML (2022)
  * [25] Lindquist, A., Picci, G.: On the stochastic realization problem. SIAM Journal on Control and Optimization 17(3), 365–389 (1979)
  * [26] Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual generation with composable diffusion models. In: European Conference on Computer Vision. pp. 423–439. Springer (2022)
  * [27] Liu, T.Y., Golatkar, A., Soatto, S.: Tangent transformers for composition, privacy and removal. arXiv preprint arXiv:2307.08122 (2023)
  * [28] Liu, T.Y., Soatto, S.: Tangent model composition for ensembling and continual fine-tuning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 18676–18686 (2023)
  * [29] Malladi, S., Wettig, A., Yu, D., Chen, D., Arora, S.: A kernel-based view of language model fine-tuning. In: International Conference on Machine Learning. pp. 23610–23641. PMLR (2023)
  * [30] Matena, M., Raffel, C.: Merging models with fisher-weighted averaging, 2021. arXiv preprint arXiv:2111.09832
  * [31] Neal, R.M.: Annealed importance sampling. Statistics and computing 11, 125–139 (2001)
  * [32] Pinkney, J.N.M.: Pokemon blip captions. https://huggingface.co/datasets/lambdalabs/pokemon-blip-captions/ (2022)
  * [33] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [34] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. pp. 8821–8831. PMLR (2021)
  * [35] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [36] Salimans, T., Ho, J.: Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 (2022)
  * [37] Schuhmann, C.: LAION-Aesthetics. https://github.com/christophschuhmann/improved-aesthetic-predictor (2022)
  * [38] Somepalli, G., Singla, V., Goldblum, M., Geiping, J., Goldstein, T.: Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems 36 (2024)
  * [39] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
  * [40] Song, Y., Durkan, C., Murray, I., Ermon, S.: Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems 34, 1415–1428 (2021)
  * [41] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)
  * [42] Tan, W.R., Chan, C.S., Aguirre, H., Tanaka, K.: Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing 28(1), 394–409 (2019). https://doi.org/10.1109/TIP.2018.2866698, https://doi.org/10.1109/TIP.2018.2866698
  * [43] Vempala, S., Wibisono, A.: Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. Advances in neural information processing systems 32 (2019)
  * [44] Vyas, N., Kakade, S., Barak, B.: Provable copyright protection for generative models. arXiv preprint arXiv:2302.10870 (2023)
  * [45] Wei, T., Guo, Z., Chen, Y., He, J.: Ntk-approximating mlp fusion for efficient language model fine-tuning (2023)
  * [46] Wortsman, M., Ilharco, G., Gadre, S.Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A.S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al.: Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In: International Conference on Machine Learning. pp. 23965–23998. PMLR (2022)
  * [47] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems 36 (2024)
  * [48] Xue, Z., Song, G., Guo, Q., Liu, B., Zong, Z., Liu, Y., Luo, P.: Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems 36 (2024)
  * [49] Yang, K.Y., Wibisono, A.: Convergence in kl and rényi divergence of the unadjusted langevin algorithm using estimated score. In: NeurIPS 2022 Workshop on Score-Based Methods (2022)
  * [50] Zancato, L., Achille, A., Ravichandran, A., Bhotika, R., Soatto, S.: Predicting training time without training. Advances in Neural Information Processing Systems 33, 6136–6146 (2020)
  * [51] Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A.M., Le, Q.V., Laudon, J., et al.: Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems 35, 7103–7114 (2022)

##  Appendix 0.A Diffusion Soup: Model Merging for Text-to-Image Diffusion
Models (Supplementary Material)

Figure 7: (Left) Diffusion Soup Outperforms All Models at All Checkpoints. We
show that optimizing over the number of SGD steps at LR = 1e-6 demonstrates
that at all checkpoints, the souping model outperforms all individual models,
as well as the combined models. The shallow dropoff in performance of the
souped model relative to the steep dropoff of individual models at larger
steps is indicative of strong robustness properties of Diffusion Soup. (Right)
Souping Outperforms the Paragon even when Optimizing the Latter for LR and
Step Size Jointly. We show that optimizing over Learning Rates and Step size
jointly yields a better Combined Model Paragon (LR = 5e-7, Num Steps = 80k)
than the original used in the main text (LR = 1e-6, Num Steps = 10k). However,
Diffusion Soup still outperforms (TIFA 85.9 $\to$ 86.2) this new Combined
Model Paragon checkpoint.

The supplementary material is organized as follows. Appendix 0.B provides
further details about hyperparameters used to conduct experiments in the
paper, alongside ablations that justify our choice of hyperparameters.
Appendix 0.C provides a comparison of Diffusion Soup to a computationally
expensive Ensemble model. Appendix 0.D provides additional examples of style
mixing using compositions of famous artistic styles. Appendix 0.E provides
additional examples of greedy souping. Finally, Appendix 0.F gives proofs for
the main propositions of the paper.

##  Appendix 0.B Choices of Hyperparameters and Ablations

The primary focus of our work is to showcase the benefits of Diffusion Soup
atop a pre-existing collection of finetuned checkpoints. Specifically, we aim
to demonstrate that regardless of the ingredients provided, souping improves
performance over any individual model, or the traditional approach of
aggregating datasets to build one model (the Combined Paragon model). In this
section, we show that regardless of the finetuned checkpoints, souping
outperforms its constituent ingredients. We further optimize learning rates to
find the strongest possible combined model paragon, to show the dominance of
the souping approach even when the paragon is hyperparameter tuned at a
disadvantage to souping.

To train our finetuned checkpoints for both the Specialist experiments and the
Aesthetic Enhancement experiments, we use the AdamW optimizer with a Learning
Rate of 1e-6. Our choice is mainly motivated by the Emu Paper [9] who
recommend a low learning rate and finetuning for up to 15k steps to prevent
model memorization and forgetting. We ablate the choice of the number of steps
in Fig. 7 (Left), and come to several conclusions. First, we show that as [9]
suggest, the dropoff point for all finetuned models begins around 15k steps,
and all models, including the Combined model peak at 10k steps. We thus use
the 10k checkpoints for all of our individual and souping experiments in the
main text. Moreover, we show that regardless of the number of steps chosen,
the souped model always outperforms the constituent ingredients and the
combined model paragon, a powerful result that is indicative of the robustness
of the souping approach. The horizontal black bar demonstrates that souping
achieves the highest value overall, a TIFA score of $86.2$, the number we
report in the main text for Uniform Souping.

We further ablate the choice of learning rate and step size for the Combined
model Paragon in Fig. 7 (Right), to ensure that souping is being compared to
the strongest possible paragon baseline. We find that a combination of an even
lower learning rate (5e-7), and an extremely large step size of 80k steps
outperforms (85.5 $\to$ 85.9) the combined model paragon at the learning rate
of 1e-6 and step size of 10k that we use in the main text. However, our souped
model at 1e-6 and step size of 10k still outperforms (85.9 $\to$ 86.2) this
new Combined model checkpoint. We thus show that our souping methodology and
findings are robust to hyperparameter tuning.

Table 3: Diffusion Soup is Competitive even with a Computationally Expensive
Ensembling Approach. We revisit the setting of Table 2 in the main text,
taking the first two aesthetic subsets AE1 and AE2, in order to compare
Diffusion Soup with an ensembling approach. As observed previously, souping
outperforms each constituent model and the combined paragon. We further show
that the performance of our model is even competitive with a computationally
costly Ensemble paragon, which computes the outputs of the denoising network
using every model at every time step. The relative closeness of the two
approaches, and the computational feasibilty of Diffusion Soup makes it a
particularly attractive choice for practical users of compartmentalization.

|  |  TIFA |  IR |  CLIP
---|---|---|---|---
Base |  SD2.1 |  85.5 |  0.30 |  0.259
SFT |  AE1 |  86.5 |  0.53 |  0.263
AE2 |  86.3 |  0.54 |  0.261
Combined (Paragon) |  AE1 & AE2 |  86.0 |  0.40 |  0.261
Ensemble (Paragon) |  AE1 & AE2 |  87.3 |  0.64 |  0.263
Souped (Ours) |  AE1 & AE2 |  86.7 |  0.60 |  0.263

##  Appendix 0.C Comparison to Ensembling

In this section we compare Diffusion Soup against the ensembling baselines
[15, 12, 26]. These methods train different models on different subsets of
data, and merge the flows—outputs of the denoising network, rather than the
weights of the network as in souping—at inference. Precisely, ensembling
approaches compute the output of the denoising network of each ingredient
model at each time-step during backward diffusion and average them using
weighted coefficients. We follow the approach in [15] which shows that the
weighted coefficients can be computed using a trained classifier.

Critically, ensembling approaches require _each ingredient model to be loaded
and run at inference time_. Therefore, they have a much larger memory
footprint, and much greater inference overhead than Diffusion Soup, which
generates a single model for use at inference time. To provide an anecdote
illustrating the difference, in this paper we run all of our experiments on a
single p4d.24xlarge EC2 instance in the AWS Cloud. Whereas our souping
experiments often aggregate 5-9 model checkpoints together, we can only
ensemble with 2 models without running into OOM errors on an A100 GPU.

We work within these limitations in Tab. 3, where we compare our method
against the ensembling baselines. We take a pre-trained Stable-Diffusion model
and fine-tune it in on two subsets of the MS-COCO dataset. Diffusion Soup is
within $1\%$ ($86.7\to 87.3$) TIFA and $93.75\%$ IR ($0.6\to 0.64$) to
ensembling. The small performance gap between the two approaches, and the
computational feasibilty and scalability of Diffusion Soup makes souping a
particularly attractive choice for practical users of compartmentalization.

##  Appendix 0.D Additional Examples of Style Mixing

We present additional zero-shot style mixing results using models fine-tuned
on _Ukiyo-e_ and _Romanticism_ images from the WikiArt dataset [42]. Fig. 8
demonstrates souping Ukiyo-e and Romanticism styles. Fig. 9 demonstrates
blending the Ukiyo-e style with Pokemon [32] and FSCOCO [8] styles. The final
row of this figure demonstrates souping _all three_ styles.

Figure 8: Diffusion Soup merges finetuned models to create hybrid artistic
styles. We apply Diffusion Soup to the SD2.1 model (_Row 1_) finetuned on
Ukiyo-e (_Row 2_) and Romanticism (_Row 3_) styles from WikiArt [42] to create
a hybrid style (_Row 4_). The results are zero-shot since we do not have
examples of the hybrid style for training.

Figure 9: Diffusion Soup merges _three_ finetuned models to create hybrid
styles. We apply Diffusion Soup to the SD2.1 model (_Row 1_) finetuned on
Ukiyo-e (_Row 2_) to create various hybrid styles. Ukiyo-e is souped with
FSCOCO (_Row 3_), Pokemon (_Row 4_) and both models (_Row 5_). Table 4: Greedy
Souping Methods Outperform all Approaches in Aesthetic Enhancement. We once
again consider the setting of the main paper’s Table 2, with data shards that
all have one task in common, aesthetic enhancement. This time, we add our
Greedy Souping approaches to Table 2 (R. refers to Reverse), showing that they
outperform all other methods in Aesthetic Enhancement. These findings
replicate the strong performance of Greedy Approaches found in aggregating
specialists in the main text’s Table 1.

| Base & SFT | Combined (Cumulative) | Souped (Cumulative)
---|---|---|---
|  TIFA |  IR |  CLIP |  TIFA |  IR |  CLIP |  TIFA |  IR |  CLIP
SD2.1 |  85.5 |  0.30 |  0.259 |  - |  - |  - |  - |  - |  -
AE1 |  86.5 |  0.53 |  0.263 |  86.5 |  0.53 |  0.263 |  86.5 |  0.53 |  0.263
AE2 |  86.3 |  0.54 |  0.261 |  86.0 |  0.40 |  0.261 |  86.7 |  0.60 |  0.263
AE3 |  86.1 |  0.47 |  0.262 |  85.7 |  0.38 |  0.261 |  86.8 |  0.60 |  0.263
AE4 |  86.2 |  0.43 |  0.260 |  86.0 |  0.39 |  0.261 |  86.8 |  0.59 |  0.263
AE5 |  85.9 |  0.45 |  0.262 |  85.6 |  0.37 |  0.261 |  86.8 |  0.59 |  0.263
Greedy Soup |  - |  - |  - |  - |  - |  - |  87.0 |  0.60 |  0.263
R. Greedy Soup |  - |  - |  - |  - |  - |  - |  86.9 |  0.59 |  0.263

##  Appendix 0.E Greedy souping for _Enhancing Image Aesthetics_

We extend the experimental section from Sec. 6.2.1 and Tab. 2 with Greedy
Souping results (see Sec. 4.2). The original cumulative findings showed that
Uniform Soup outperforms its ingredients in a cumulative sense, i.e. as the
number of checkpoints grow from AE1 all the way to AE1-AE5. Our greedy
approaches strategically select a subset of these checkpoints to show that
performance can be further optimized in TIFA score. Identical to the setting
with Specialists, Greedy Souping outperforms all other souping approaches, and
both Greedy Souping and Reverse Greedy Souping are the two highest performing
approaches. Our strong performance demonstrates that Greedy Souping offers
benefits when merging models trained on a common task (in this case aesthetic
enhancements) as well as aggregating specialists (see Tab. 1).

##  Appendix 0.F Proofs

We restate each proposition for convenience prior to providing their proof.

###  0.F.1 Proposition 1

###### Proposition 1

(Geometric Mean) Let $\nabla_{x_{t}}\log p^{(i)}(x_{t})$ be the marginal score
(in Eq. 2) where $x_{t}=\gamma_{t}x_{0}+\sigma_{t}\epsilon$, with $x_{0}\sim
p^{(i)}(x_{0})$ and $\epsilon\sim\mathcal{N}(0,I)$. Let
$\epsilon_{w_{i}}(x_{t},t,y)$ be a neural network (with sufficient capacity)
trained to match $\nabla_{x_{t}}\log p^{(i)}(x_{t})$. Under certain conditions
on the sampling procedure Eq. 2, using $(1/n)\sum_{i=1}^{n}\nabla_{x_{t}}\log
p^{(i)}(x_{t})$ generates samples from
$Z^{-1}\big{(}\Pi_{i}p^{(i)}(x_{0})\big{)}^{1/n}$. Furthermore, using Eq. 4,
$\epsilon_{\text{soup}}$ with $k_{i}=1/n$ generates samples from
$Z^{-1}\big{(}\Pi_{i}p^{(i)}(x_{0})\big{)}^{1/n}$.

Proof Let
$p_{\text{geometric}}=Z^{-1}\big{(}\Pi_{i}p^{(i)}(x_{0})\big{)}^{1/n}$ be the
geometric mean of the initial set of distributions provided to us. The
marginal distribution at time $t$ corresponding to the geometric mean is given
by $p(x_{t})=\int
Z^{-1}\big{(}\Pi_{i}p^{(i)}(x_{0})\big{)}^{1/n}p(x_{t}|x_{0})dx_{0}$. Sampling
from this distribution requires access to the score of the marginal, which is
usually difficult to obtain. This score is given by:

$\nabla_{x_{t}}\log p(x_{t})=\nabla_{x_{t}}\log\int
Z^{-1}\big{(}\Pi_{i}p^{(i)}(x_{0})\big{)}^{1/n}p(x_{t}|x_{0})dx_{0}$

where the integral inside the logarithm is difficult to estimate. However, we
can show that we can construct a simple score function which is easy to
estimate and can generate samples from $p_{\text{geometric}}$:

$\nabla_{x_{t}}\log p^{\prime}(x_{t})=(1/n)\sum_{i}\nabla_{x_{t}}\log\int
p(x_{t}|x_{0})p^{(i)}(x_{0})dx_{0}$

$\nabla_{x_{t}}\log p^{\prime}(x_{t})$ can be used for sampling because of the
fact that it generates a sequence of distributions where its equal to
$\nabla_{x_{t}}\log p(x_{t})$ at $t=0$, and $\mathcal{N}(0,I)$ at $t=T$. Thus
using $\nabla_{x_{t}}\log p^{\prime}(x_{t})$ we can create a sequence of
distributions which generates a sample from $p_{\text{geometric}}$ as
$t\rightarrow 0$ during backward diffusion using Eq. 2. Note that this
sequence of distributions is different from the sequence of distributions
generated by the forward diffusion process. Convergence of langevin based MCMC
(using Eq. 2 for sampling) is a very well studied problem [49, 6, 43, 31],
showing that using Eq. 2 indeed converges to stationary distribution given by
$p_{\text{geometric}}$ in this case. Furthermore,[49] in Theorem 1 shows that
sampling using Langevin dynamics (Eq. 2) with a sufficiently powerful score
estimator (in our case a diffusion model) generates samples from the initial
distribution, $p_{\text{geometric}}$. This proves the first part of the
result. The second part of the proof is a mere application of Eq. 4 and Eq. 5
i.e. we replace each $\nabla_{x_{t}}\log p^{(i)}(x_{t})$ with
$-\epsilon_{w_{i}}(x_{t},t,y)$. More precisely,

$\nabla_{x_{t}}\log
p(x_{t})=(1/n)\sum_{i}-\epsilon_{w_{i}}(x_{t},t,y)=-\epsilon_{\text{soup}}$
(8)

###  0.F.2 Proposition 2

###### Proposition 2

(Arithmetic Mean) Let $\nabla_{x_{t}}\log p^{(i)}(x_{t})$ be the marginal
score (in Eq. 2) where $x_{t}=\gamma_{t}x_{0}+\sigma_{t}\epsilon$, with
$x_{0}\sim p^{(i)}(x_{0})$ and $\epsilon\sim\mathcal{N}(0,I)$. Let
$\epsilon_{w_{i}}(x_{t},t,y)$ be a neural network (with sufficient capacity)
trained to match $\nabla_{x_{t}}\log p^{(i)}(x_{t})$. Sampling using
$\sum_{i=1}^{n}\lambda_{i}\frac{p^{(i)}(x_{t})}{p(x_{t})}\nabla_{x_{t}}\log
p^{(i)}(x_{t})$ with Eq. 2 generates samples from
$\sum_{i}\lambda_{i}p^{(i)}(x_{0})$, where
$p(x_{t})=\sum_{i}\lambda_{i}p^{(i)}(x_{t})$, $\sum_{i}\lambda_{i}=1$.
Furthermore, using Eq. 4, $\epsilon_{\text{soup}}$ with
$k_{i}=\lambda_{i}\frac{p^{(i)}(x_{t})}{p(x_{t})}$ generates samples from
$\sum_{i}\lambda_{i}p^{(i)}(x_{0})$.

Proof Let $\nabla_{x_{t}}\log p(x_{t})$ be the score the mixture of
distributions $p(x_{t})=\sum_{i}\lambda_{i}p^{(i)}(x_{t})$. We show that the
score of the mixture is a convex combination of the individual scores:

$\displaystyle\nabla_{x_{t}}\log p(x_{t})$
$\displaystyle=\nabla_{x_{t}}\log\sum_{i}\lambda_{i}p^{(i)}(x_{t})$
$\displaystyle=\dfrac{1}{\sum_{i}\lambda_{i}p^{(i)}(x_{t})}\sum_{i}\lambda_{i}\nabla_{x_{t}}p^{(i)}(x_{t})$
$\displaystyle=\dfrac{1}{\sum_{i}\lambda_{i}p^{(i)}(x_{t})}\sum_{i}\lambda_{i}\dfrac{p^{(i)}(x_{t})}{p^{(i)}(x_{t})}\nabla_{x_{t}}p^{(i)}(x_{t})$
$\displaystyle=\dfrac{1}{\sum_{i}\lambda_{i}p^{(i)}(x_{t})}\sum_{i}\lambda_{i}p^{(i)}(x_{t})\nabla_{x_{t}}\log
p^{(i)}(x_{t})$
$\displaystyle=\sum_{i}\dfrac{\lambda_{i}p^{(i)}(x_{t})}{\sum_{i}\lambda_{i}p^{(i)}(x_{t})}\nabla_{x_{t}}\log
p^{(i)}(x_{t})$ $\displaystyle\implies\nabla_{x_{t}}\log p(x_{t})$
$\displaystyle=\sum_{i}\dfrac{\lambda_{i}p^{(i)}(x_{t})}{\sum_{i}\lambda_{i}p^{(i)}(x_{t})}\nabla_{x_{t}}\log
p^{(i)}(x_{t})$

Hence, we show the first part of the proof. The second part of the proof is a
mere application of Eq. 4 and Eq. 5 i.e. we replace each $\nabla_{x_{t}}\log
p^{(i)}(x_{t})$ with $-\epsilon_{w_{i}}(x_{t},t,y)$:

$\displaystyle\nabla_{x_{t}}\log p(x_{t})$
$\displaystyle=\sum_{i}\dfrac{\lambda_{i}p^{(i)}(x_{t})}{\sum_{i}\lambda_{i}p^{(i)}(x_{t})}(-\epsilon_{w_{i}}(x_{t},t,y))$
$\displaystyle=-\epsilon_{\text{soup}}$

where we use Eq. 5 to show the previous equality with
$k_{i}=\dfrac{\lambda_{i}p^{(i)}(x_{t})}{\sum_{i}\lambda_{i}p^{(i)}(x_{t})}$.

###  0.F.3 Proposition 3

###### Proposition 3

($\varepsilon$-NAF for Diff. Soup) Let
$\epsilon_{\text{soup}}(x_{t},t,y)\triangleq\epsilon_{\frac{1}{2}(w_{1}+w_{2})}(x_{t},t,y)$
be the soup model obtained by training $w_{i}$ on $D_{i}$, such that
$D=D_{1}\cup D_{2}$, and $D_{1}\cap D_{2}=\varnothing$. Then, under certain
conditions sampling, using $\epsilon_{\text{soup}}(x_{t},t,y)$ satisfies
$\varepsilon$-NAF for some $\varepsilon$ which depends only on $D_{1}$ and
$D_{2}$.

Proof [44] shows that given two generative models $p_{1}(x|y),p_{2}(x|y)$,
sampling from $\sqrt{p_{1}(x|y)p_{2}(x|y)}/Z$ produces data which is NAF with
respect to some protected samples in the training data. We need to show that
souping generates samples from $\sqrt{p_{1}(x|y)p_{2}(x|y)}/Z$. Note that this
directly follows from Proposition 1 for the case $n=2$.


1 School of Statistics, Capital University of Economics and Business, Beijing,
China
2 School of EECS, University of Ottawa, Ottawa, Canada
† These authors contributed equally.

# Explorers at #SMM4H 2023: Enhancing BERT for Health Applications through
Knowledge and Model Fusion

Xutong Yue1,† Xilai Wang2,† Yuxin He1 Zhenkun Zhou1

<EMAIL_ADDRESS><EMAIL_ADDRESS>{heyuxin<EMAIL_ADDRESS>
## Abstract

An increasing number of individuals are willing to post states and opinions in
social media, which has become a valuable data resource for studying human
health. Furthermore, social media has been a crucial research point for
healthcare now. This paper outlines the methods in our participation in the
#SMM4H 2023 Shared Tasks, including data preprocessing, continual pre-training
and fine-tuned optimization strategies. Especially for the Named Entity
Recognition (NER) task, we utilize the model architecture named W2NER1 that
effectively enhances the model generalization ability. Our method achieved
first place in the Task 3. 111This paper has been peer-reviewed and accepted
for presentation at the #SMM4H 2023 Workshop.

##  1 Introduction

The Social Media Mining for Health Applications (SMM4H) Shared Tasks aim to
address natural language processing (NLP) challenges of using social media
data for health informatics, including informal, colloquial expressions,
misspellings, noise, data sparsity, ambiguity, and multilingual posts 2 . We
participated in three tasks in SMM4H 2023 (i.e. Task 1, Task 3, Task 4). Task
1 and Task 4 focus on COVID-19 diagnosis in self-reported English tweets and
self-reported social anxiety disorder diagnosis posted in Reddit. Task 3
concentrates on detecting and extracting COVID-19 symptoms in Latin American
Spanish tweets description.

The dataset of Task 1 contain texts from Twitter that self-report the COVID-19
diagnosis (labeled as ‘1’) or not (labeled as ‘0’). The size of the training
set, validation set, and test set are 7600, 400, 10000. Task 4 contains 8117
posts from users aged 12 to 25. Positive cases (labeled as ‘1’) represent
self-reported or probable social anxiety disorder diagnoses, while negative
cases (labeled as ‘0’) include users without a diagnosis or with uncertain
diagnostic status. The sizes of the training set, validation set, and those of
test set are 6090, 680, 1347. Task 3 focuses on the detection and extraction
of COVID-19 symptoms in tweets written specifically in Latin American Spanish,
includes both personal self-reports and third-party mentions of symptoms.
There are 6021 of the training data, 1979 for validation, and 2150 for
testing.

In the paper, we present methodologies for addressing three tasks. For the
classification tasks, we adopt three pre-trained language models (PLMs) as our
backbones: RoBERTa-base 3, vinai-bertweet 4, and CPM-RoBERTa. For NER task, we
select bert-base-spanish-wwm-cased-xnli 5, BETO 6, and BETO_NER 7. We also
propose tricks to mitigate the challenges existing in dataset, including
category imbalance, domain disparities, and text noise. Our approach achieves
state-of-the-art results in Task 3 and surpasses the median performance in
Task 1.

##  2 Methods

Methods used for task 1 and 4 consist of three main components: continual pre-
processing, continual pre-training, and model fine-tuning. In Task 3, based on
methods for Task 1 and 4, we use additiaonally the W2NER model architecture.

### Pre-processing

In the provided dataset, it contains a large number of emojis and usernames,
which lacks valid textual semantic information. To enhance the model’s focus
on textual semantics, we preprocess the data by replacing emojis with
corresponding emotion description, and usernames with ‘@user’. Hashtags
contains some information sometimes, we preprocess ‘#hashtag’ by separating
‘#’ and the text follows.

### Continual Pre-training

The purpose of continual pre-training is to obtain pre-trained models that are
more applicable to specific industries or domains, aiming to improve the
effectiveness of the models in downstream tasks 8. Pretrained models are
usually trained on large-scale generalized data, and thus may not have a deep
enough understanding of some task-specific data. To solve the domain bias
problem, we continue with domain-adaptive pretraining using unlabeled data
from all the related tasks to optimize the model for task-specific data 9.
Specifically, for the classification task, we select all relevant COVID-19
self-reported English tweets data, and anxiety self-reported data to continue
the task-adaptive pretraining of the three pretrained models with a learning
rate of 3e-5 and 20 epochs. For the NER task, we use the training and
validation data for continual pre-training, with the learning rate of 3e-5 and
30 epochs. The effects of continual pre-training are quite unstable, sometimes
leading to positive impacts and other times resulting in negative influences,
which are largely correlated with the data used for pre-training. Fortunately,
in the 3 tasks we conducted, continual pre-training yielded positive results.

### W2NER Model Architecture

In Task 3, rather than employing a conventional model architecture, we adopt
the state-of-the-art W2NER model architecture. The W2NER architecture captures
neighboring relations between entity words using two distinct types of
relations(the Next-Neighboring-Word, NNW, and the Tail-Head-Word, THW).
Building upon this approach, the creator of the W2NER model introduced a
neural framework that represents unified Named Entity Recognition (NER) as a
2D grid of word pairs. Leveraging the W2NER architecture, we achieved the top
ranking in Task 3.

### Optimization Strategy

In Task 1 and 4, we implemented all 4 optimization strategies below. For Task
3, only Multi-Model Fusion Voting is employed.

Focal Loss: Classification tasks exhibit noticeable category imbalance. The
Task 1 has a positive example proportion of 17.6%, and the Task 4 has 37.8%.
To address the problem, we utilize focal loss to adjust sample weights during
training, emphasizing hard samples 10. Specifically, for binary
classification, the traditional cross-entropy loss function is expressed as:

$CE(p,y)=-(y\cdot log(p)+(1-y)\cdot log(1-p))$ (1)

Focal loss introduces a modifying factor $\gamma$. The modified loss function
is:

$FL(p,y)=-\alpha\cdot(1-p)^{\gamma}\cdot log(p)$ (2)

We also tried to randomly replicate samples in a few classes using
oversampling to make the training data more balanced between positive and
negative samples.

Single-model Fusion Voting: In Task 1 and 4, we employ five-fold cross-
validation, dividing the training set into five subsets. Four subsets are used
for training, and one is for development. Using the same model structure and
initial parameters, we train five models on the subsets. The models’ results
are combined through average voting and applied to the test data.

Parameter Average Generalization: Overfitting occurs when a model learns
excessive details and noises from the training data, resulting in decreased
generalization performance on the real data. To enhance the model’s robustness
and reduce training data oscillations caused by noise and variations, we
employ parameter average generalization. Specifically, exponential sliding
average (EMA): $\theta_{t}$ is the model weight at time $t$.The original
weights are still used during the training process and shadow weights are
calculated after a certain number of steps, which are used in the testing
phase to replace the weights in the neural network:

$v_{n}=\alpha\cdot v_{n-1}+(1-\alpha)\theta_{n}$ (3)
$v_{t}=\frac{v_{t}}{(1-\beta^{t})}$ (4)

Multi-Model Fusion Voting: In order to further improve the model
generalization ability and achieve the complementary advantages of different
models under the unified task, we use multi-model fusion voting in Task 1, 3
and 4 to get the final prediction results of different model prediction
results for the same task through Mean-Pooling. For Task 3, bert-base-spanish-
wwm-cased-xnli, BETO, and BETO_NER are used for Multi-Model Fusion Voting.

##  3 Results

### Task 1 and Task 4

For each model, we set batch size to 32, epoch to 30, learning rate to 3e-5
and weight decay to 0.01 for bert’s internal parameters, set learning rate to
3e-4 and weight decay to 0 for the classification layer parameters. the
optimizer uses AdamW11. For the unbalanced samples, we use focal loss with a
weight parameter of 0.25, and the adjustment factor of 2. The experiments’
results on validation are shown in Table 1. Continual pre-training is
beneficial to allow the pre-trained model to learn some domain-specific
information, in order to test the effectiveness of continual pre-training to
improve the performance of the model, we present the results of two versions,
our F1-scores are shown in Table 2. For each task, we present model
predictions that show the best performance in the validation set, our model
exceeds the mean of Task 1 by 0.06. The results on Task 4 are below the
median.

Table 1: Table 1:F1-scores of classification tasks. Original means using the original training dataset, Oversampling means using the oversampled set, and Focal_Loss means using only focal_loss without oversampling. Task | Original | Oversampling | Focal Loss
---|---|---|---
#1 | 0.806 | 0.885 | 0.870
#4 | 0.648 | 0.687 | 0.709
Table 2: F1-scores of classification tasks, Original means that Continual Pre-training is not used. Task | Original | Continual Pre-training
---|---|---
#1 | 0.865 | 0.872
#4 | 0.677 | 0.695

### Task 3

Regarding each model, we configured the batch size as 12, the number of epochs
as 30, and the learning rate for BERT’s internal parameters as 1e-5.
Additionally, we designated a learning rate of 1e-3 and a weight decay of 5e-3
for the remaining parameters. We also employ continual pre-training and multi-
model fusion voting in Task 3. For the continual pre-training, the number of
epochs is 30, learning rate is 3e-5. As a result, on the test set, our
precision is 0.94, recall is 0.93, F1-score is 0.94. Our F1-score is the
highest among all the submissions for Task 3.

##  4 Conclusion

The SMM4H 2023 Shared Tasks aim to extract and understand users’ opinions
about healthcare from social media. We propose a deep learning model to
tackles category imbalance, domain bias and text noise. Due to time
constraints, we were unable to address all issues, such as misspellings and
informal or colloquial expressions in the Task 3. For instance, during symptom
recognition, the model successfully identifies the term ‘gripe; but overlooks
variations like ‘gripes’ and ‘gripa’. Future research that successfully
resolves this challenge has the potential to achieve the model’s performance.

## Acknowledgements

Xutong Yue, Yuxin He and Zhenkun Zhou were supported by the R&D Program of
Beijing Municipal Education Commission, Grant No. KM202210038002.

## References

  * 1 Li J, Fei H, Liu J, Wu S, Zhang M, Teng C, et al.  Unified named entity recognition as word-word relation classification.  In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36; 2022. p. 10965-73.
  * 2 Klein AZ, Banda JM, Guo Y, Flores Amaro JI, Rodriguez-Esteban R, Sarker A, Schmidt AL, Xu D, Gonzalez-Hernandez G.  Overview of the eighth Social Media Mining for Health Applications (#SMM4H) Shared Tasks.  Proceedings of the Eighth Social Media Mining for Health Applications (#SMM4H) Workshop and Shared Task. 2023.
  * 3 Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, et al.  Unsupervised Cross-lingual Representation Learning at Scale.  In: Annual Meeting of the Association for Computational Linguistics; 2019\.
  * 4 Nguyen DQ, Vu T, Nguyen AT.  BERTweet: A pre-trained language model for English Tweets.  In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations; 2020. p. 9-14.
  * 5 Recognai. bert-base-spanish-wwm-cased-xnli; 2021.
  * 6 Cañete J, Chaperon G, Fuentes R, Ho JH, Kang H, Pérez J.  Spanish pre-trained bert model and evaluation data.  arXiv preprint arXiv:230802976. 2023.
  * 7 Romero M. bert-spanish-cased-finetuned-ner; 2020.
  * 8 Gururangan S, Marasović A, Swayamdipta S, Lo K, Beltagy I, Downey D, et al.  Don’t stop pretraining: Adapt language models to domains and tasks.  arXiv preprint arXiv:200410964. 2020.
  * 9 Fu J, Gan Z, Li Z, Li S, Sui D, Chen Y, et al.  CASIA at SemEval-2022 Task 11: Chinese named entity recognition for complex and ambiguous entities.  In: Proceedings of the 16th international workshop on semantic evaluation (SemEval-2022); 2022. p. 1518-23.
  * 10 Lin TY, Goyal P, Girshick R, He K, Dollár P.  Focal loss for dense object detection.  In: Proceedings of the IEEE international conference on computer vision; 2017. p. 2980-8.
  * 11 Kingma DP, Ba J.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:14126980. 2014.


WordEmbedding-Static(GloVe)     0.9642

4*TCN     WordEmbedding-Random     0.9762

WordEmbedding-Static (Word2Vec)     0.9784

WordEmbedding-Dynamic (Word2Vec)     0.9791

3*Ensemble (CNN-BiGRU)     WordEmbedding-Random     0.9722

WordEmbedding-Static (Word2Vec)     0.9763

WordEmbedding-Dynamic (Word2Vec)     0.9768

Performance of Classifiers After Paraphrasing Spam Messages (Paraphrasing Attack). Which dataset was used in this attack?

Feature Model     Classifier     3cBefore Paraphrasing     3cAfter Paraphrasing

Precision     Recall     Accuracy
Precision     Recall     Accuracy

3*BoW     TCSVM     1.00     0.86     0.86     1.00     0.58     0.58

OCSVM     1.0     0.80     0.80     1.00     0.68     0.68

PU     1.00     0.88     0.88     1.00     0.7     0.7

3*Bigram/ TF-IDF     TCSVM     1.00     0.62     0.62     1.00     0.06     0.06

OCSVM     1.0     0.48     0.48     1.00     0.26     0.26

PU     1.0     0.76     0.76     1.0     0.2     0.2

3*Trigram/ TF-IDF     TCSVM     1.00     0.44     0.44     1.00     0.04     0.04

OCSVM     1.0     0.58     0.58     1.0     0.72     0.72

PU     1.0     0.38     0.38     1.0     NA     NA

3*5-gram/ TF-IDF     TCSVM     1.00     0.38     0.38     1.00     0.00     0.00

OCSVM     1.0     0.78     0.78     1.0     0.96     0.96

PU     1.0     0.36     0.36     1.0     0.04     0.04

3*Word2Vec     TCSVM     1.00     0.9     0.9     1.00     0.82     0.82

OCSVM     1.0     0.68     0.68     1.0     0.68     0.68

PU     1.0     0.96     0.96     1.0     0.98     0.98

fastText     fastText     0.98     0.98     NA     0.78     0.78     NA

§ DISCUSSION

In this section, we summarize the key observations that we found while evaluating the aforementioned ML models and shed some light on further work and challenges in this area. In the previous section, comprehensive evaluation of different ML approaches and robustness assessment via adversarial attacks in SMS classification enable readers to have a faster and better understanding of this aspect. Two sets of experiment were performed in order to assess the effectiveness of the state of the art ML models.
In the first set of experiment, thirty two shallow ML and DL models were trained to evaluate their performance in classifying spam and legitimate messages. Moreover, we investigated the effect of the super dataset in training an efficient model, since previous research [refs] claimed that significant performance improvements can be achieved by incorporating large corpora and diverse set of examples in the dataset. The spam detection accuracy of most of the ML models improved over the super dataset in comparison to the baseline performance when they were trained over the consolidated dataset (as highlighted in the Figure XXXXX, showing accuracy of the models trained with the consolidated and the super dataset each, only highest accuracy of the model is shown).

Traditional ML Models

SNN & DL Models

Performance Comparison of Models Trained with Super and Consolidated Datasets

One of the most important aspect of this research is to explore how adversarial attacks can be used to target shallow ML and DL based SMS spam models by generating adversarial samples. A second set of experiment was carried out for this purpose and the aforementioned DL models were assessed for robustness using adversarial evaluation. Overall, the classification performance of both the ML and DL models decreased significantly when adversarial samples were present. Both the shallow ML and DL models are susceptible to various set of adversarial attacks.

To summarize, any model built on top of semantic based word embedding causes the model to perform well, both in terms of accuracy as well as robustness. The results indicate that transformer based deep learning models can increase the model robustness with a good margin. Overall, the DistilBERT model outperforms all other evaluated classifiers in terms of performance accuracy and robustness. However, FastText has also performed well and is therefore a good alternative to deep learning models especially in scenarios where low computation and reduce training time is desired. Furthermore, currently all of the ML/ DL models are vulnerable to some form of adversarial text attacks. Therefore, there is a need for a robust model for SMS spam detection with high performance accuracy.

§ FUTURE WORK & CHALLENGES

We hereby suggest that the future of SMS spam filters lies in deep learning. The shallow ML finds it very hard to mine adequately-represented features because to the limitations that characterized such algorithms. Deep learning have been applied to solve representation problem by creating several naive features to represent a complicated concept <cit.>. Deep learning is far more effective in solving the problem of spam SMS because as number of available training data is increasing, the effectiveness and efficiency of deep learning becomes more pronounced. Deep learning models have the capacity to solve sophisticated problems by using intricate and huge models. Thus, they exploit the computational power of modern CPUs and GPUs (becoming cheaper day by day and online/ cloud base availability of resources). However, despite the huge success of deep learning in solving many problems, it has been proved that similar to traditional ML, the DL models are also susceptible to adversarial examples. Therefore, the adversarial deep learning technique is a great method that is yet to be explored in SMS spam filtering.

Further research work need to be conducted to tackle the fact that SMS spam filtering is a concept drift problem. As such, while the researchers are trying to increase the accuracy of the spam model, the spammers are also evolving and trying to surpass the efficiency of the spam filters. It becomes very important to develop more efficient techniques and aggregate/ report the latest spam campaigns in a public dataset (have as much representative data as possible) that will adequately handle the trend or progression in spam features that makes them to evade many spam filters undetected. Summarily, the open research problems in SMS spam filtering are itemized below:

* Development of a fine-tuned robust SMS spam filters with high accuracy and robustness.

* Development of an effective strategy to handle the threats to the security of the ML models. Such an attack can be white box or black box, targeted or non-targeted attack.

§ CONCLUSION