# Improving Consistency Models with Generator-Induced Coupling

Thibaut Issenhuth
Criteo AI Lab, Paris, France
<EMAIL_ADDRESS>
&Ludovic Dos Santos
Criteo AI Lab, Paris, France
<EMAIL_ADDRESS>
&Jean-Yves Franceschi
Criteo AI Lab, Paris, France
<EMAIL_ADDRESS>
&Alain Rakotomamonjy
Criteo AI Lab, Paris, France
<EMAIL_ADDRESS>
###### Abstract

Consistency models are promising generative models as they distill the multi-
step sampling of score-based diffusion in a single forward pass of a neural
network. Without access to sampling trajectories of a pre-trained diffusion
model, consistency training relies on proxy trajectories built on an
independent coupling between the noise and data distributions. Refining this
coupling is a key area of improvement to make it more adapted to the task and
reduce the resulting randomness in the training process. In this work, we
introduce a novel coupling associating the input noisy data with their
generated output from the consistency model itself, as a proxy to the
inaccessible diffusion flow output. Our affordable approach exploits the
inherent capacity of consistency models to compute the transport map in a
single step. We provide intuition and empirical evidence of the relevance of
our generator-induced coupling (GC), which brings consistency training closer
to score distillation. Consequently, our method not only accelerates
consistency training convergence by significant amounts but also enhances the
resulting performance. The code is available at:
https://github.com/thibautissenhuth/consistency_GC.

##  1 Introduction

Diffusion and score-based models (Ho et al., 2020; Song et al., 2021; Karras
et al., 2022) have emerged as state-of-the-art generative models for image
generation. Since they are costly to use at inference time, as they require
several neural function evaluations, many distillation techniques have been
explored (Salimans and Ho, 2022; Meng et al., 2023; Sauer et al., 2023). A
most remarkable approach is the one coined as consistency models (Song et al.,
2023; Song and Dhariwal, 2024). Consistency models lead to high-quality one-
step generators, that can be trained either by distillation of a score-based
model, or as standalone generative models. In this paper, we study consistency
models trained without a pre-trained score model.

At the core of the training of consistency models, an independent coupling
(IC) between data and noise distribution is used. Indeed, when training a
consistency model without a pre-trained score model, any point from the data
distribution is matched to any point from the noise distribution in order to
construct trajectories. However, Pooladian et al. (2023) identified those as a
potential issue in flow matching models (Lipman et al., 2023). Notably, the
authors demonstrated that IC leads to high variance of the gradient estimator
of such generative models. They resort to batch coupling with optimal
transport (OT) tools and prove that it improves performance. Dou et al. (2024)
show that this approach can successfully be adopted in consistency models, and
we actually confirm this in our experiments. However, discrete OT has its
flaws. Most notably, it converges slowly to the true Wasserstein distance in
high dimension, and increasing the size of the coupling sets leads to
increased running time, since solvers (e.g. Hungarian matching algorithm)
typically have a cubic complexity.

Consistency models map any point on a given reverse diffusion trajectory to
the initial data point by minimizing the distance between the outputs of
adjacent points on the trajectory. Training a consistency model in such a
setting relies on constructing trajectories from an IC between data and noise
distributions, and selecting pairs of intermediate points from each single
trajectory (Song et al., 2023). Drawing inspiration from Pooladian et al.
(2023), we aim at constructing better couplings between data and noise
distributions. Instead of using discrete OT tools, we take another direction
and propose to use the generator itself to construct trajectories. From an
intermediate point computed from an IC, we let the consistency model predict
the corresponding endpoint, supposedly close to the data distribution. This
predicted endpoint is used to construct a new coupling, from which two points
are selected to train the consistency model itself. Our approach is
illustrated in Figure 1. Our claim is that this approach provides couplings
that are better aligned to the actual diffusion updates and thus approximate
better the true loss function of a consistency model. As shown in our
experimental section, this procedure results in faster convergence and often
leads to better performance of the resulting model.

Figure 1: We leverage the current consistency model predictions to compute a
new coupling used for its own training. Using the original independent
coupling (IC) $(x,z)$ and intermediate points $(x_{i},x_{i+1})$ we derive a
generator-induced coupling (GC) $(f_{\theta}(x_{i},\sigma_{i}),z)$ and
intermediate points $(\tilde{x}_{i},\tilde{x}_{i+1})$. In purple, we
illustrate that using generator-induced trajectories instead of the usual
independent trajectories results in pairs of points closer to the ideal PF-ODE
update of the corresponding point at $i+1$.

Let us summarize our contributions below:

  * •

We propose a novel type of coupling, denoted generator-induced coupling (GC),
that allows to draw generator-induced trajectories used to construct pairs of
points for the training of consistency model. We carefully identify and select
the main design components of our method through experiments.

  * •

We provide insights into the advantages of this coupling on both synthetic and
image datasets. Most notably, we show that trajectories drawn from GC are
closer to ideal trajectories defined by a pre-trained score model than
standard trajectories drawn from IC.

  * •

We experimentally demonstrate the benefits of this coupling in image
generation benchmarks. When combined with standard trajectories from IC, GC
leads to faster convergence and often improves the performance of the
generative model.

##  2 Preliminaries

#### Notation.

We consider an empirical data distribution $p_{\star}$ and noise distribution
$p_{z}$ (e.g. Gaussian) defined on $\mathds{R}^{d}$. We note $q$ a joint
distribution of samples from $p_{\star}$ and $p_{z}$. We use a distance
function $\mathcal{D}$ to measure the distance between two points from
$\mathds{R}^{d}$. $\mathrm{sg}$ denotes the stop-gradient operator.

In consistency models, we consider a discrete formulation of diffusion models
with $N$ intermediate timesteps. A vector from an intermediate timestep can be
sampled as $x_{i}=\alpha_{i}x+\sigma_{i}z$, where $x\sim p_{\star}$, $z\sim
p_{z}$, and $(\alpha_{i})$ (resp. monotonous $(\sigma_{i})$) are data (resp.
noise) schedules. In the commonly used EDM process (Karras et al., 2022),
$\alpha_{i}=1$ and can thus be omitted. We denote
$(p_{\sigma_{i}})_{i\in[0,N]}$ the corresponding intermediate noisy
distributions between $p_{\sigma_{0}}\approx p_{\star}$ and
$p_{\sigma_{N}}\approx p_{z}$, such that $x_{i}\sim p_{\sigma_{i}}$.

###  2.1 Score-Based Diffusion Models

Score-based diffusion models (Ho et al., 2020; Song et al., 2021) can generate
data from noise via a multi-step process consisting in numerically solving
either a stochastic differential equation (SDE), or an equivalent ordinary
differential equation (ODE) called the probability flow ODE (PF-ODE). Although
SDE solvers generally exhibit superior sampling quality, the PF-ODE has
desirable properties. Notably, it defines a deterministic map from noise to
data. In this context, the diffusion process based on the EDM formulation
(Karras et al., 2022) with $\alpha_{i}=1$ defines it as follows:

$\operatorname{d\\!}x=-\sigma\nabla_{x}\log
p_{\sigma}(x)\operatorname{d\\!}\sigma,$ (1)

where $\nabla_{x}\log p_{\sigma}$, a.k.a. the score function, can be
approximated with a neural network $s_{\Phi}(x,\sigma)$ (Vincent, 2011; Song
and Ermon, 2019).

###  2.2 Consistency Models

Numerically solving the PF-ODE is costly because of multiple expensive
evaluations of the score function. To alleviate this issue, Song et al. (2023)
proposed to leverage the deterministic property of the PF-ODE in a
_consistency model_ $f_{\theta}$, learning the output map of the PF-ODE, such
that:

$f_{\theta}(x_{i},\sigma_{i})=x_{0},$ (2)

for all $(x_{i},\sigma_{i})\in\mathds{R}^{D}\times[\sigma_{0},\sigma_{N}]$
that belong to the trajectory of the PF-ODE ending at $(x_{0},\sigma_{0})$.

Equation 2 is equivalent to _(i)_ enforcing the boundary condition
$f_{\theta}(x_{0},\sigma_{0})=x_{0}$ and _(ii)_ ensuring that $f_{\theta}$ has
the same output for any two samples of a single PF-ODE trajectory—the
consistency property. _(i)_ is naturally satisfied by the following model
parametrization:

$f_{\theta}(x_{i},\sigma_{i})=c_{\text{skip}}(i)x_{i}+c_{\text{out}}(i)F_{\theta}(x_{i},\sigma_{i})$
(3)

where $c_{\text{skip}}(0)=1$, $c_{\text{out}}(0)=0$, and $F_{\theta}$ is a
neural network. _(ii)_ is achieved by minimizing the distance between the
outputs of two same-trajectory consecutive samples with the consistency loss:

${L}_{\text{consistency-
distillation}}(\theta)=\mathbb{E}_{i\sim\mathcal{U}([0,N]),x_{i+1}\sim
p_{\sigma_{i+1}}}\Big{[}\mathcal{D}\Big{(}\mathrm{sg}(f_{\theta}(x_{{i}},\sigma_{i})\big{)},f_{\theta}(x_{{i+1}},\sigma_{i+1})\Big{)}\Big{]},$
(4)

where $x_{i}$ is computed by discretizing the PF-ODE of Equation 1 with the
Euler scheme as follows:

$x_{{i}}=x_{{i+1}}-(\sigma_{i}-\sigma_{i+1})\sigma_{i+1}\nabla_{x_{{i+1}}}\log
p_{\sigma_{i+1}}(x_{{i+1}}).$ (5)

The latter loss can be directly used to distill a score model
$s_{\Phi}(x_{{i+1}},\sigma_{i+1})\approx\nabla\log
p_{\sigma_{i+1}}(x_{{i+1}})$ into $f_{\theta}$. In the case of consistency
training, Song et al. (2023) circumvent the lack of score function by noticing
that $\nabla\log
p_{\sigma_{i+1}}(x_{{i+1}})=\mathds{E}\Big{[}\frac{x_{{i+1}}-x}{\sigma_{i+1}^{2}}|x_{{i+1}}\Big{]}$.
This results in selecting pairs of points $x_{i}=x+\sigma_{i}z$ and
$x_{i+1}=x+\sigma_{i+1}z$ with the same _independent_ $(x,z)\sim
q_{\mathrm{I}}=p_{\star}(x)p_{z}(z)$:

${L}_{\text{consistency}}(\theta)=\mathbb{E}_{(x,z)\sim
q_{\mathrm{I}},x_{i}=x+\sigma_{i}z,x_{i+1}=x+\sigma_{i+1}z}\Big{[}\mathcal{D}\Big{(}\mathrm{sg}(f_{\theta}(x_{{i}},\sigma_{i})\big{)},f_{\theta}(x_{{i+1}},\sigma_{i+1})\Big{)}\Big{]},$
(6)

which provably converges to the distillation loss of Equation 4 when
$N\to\infty$.

##  3 Data-Noise Coupling

As seen in Section 2.2, consistency models compute vectors from intermediate
timesteps through an IC $q_{\mathrm{I}}=p_{\star}(x)p_{z}(z)$ of data and
noise, in a similar fashion to flow matching (Lipman et al., 2023). This IC
takes root in score-based diffusion models which have inspired these more
recent approaches (Kingma and Gao, 2024). Indeed, approximating the score
function with a neural network entails sampling data points perturbed by a
Gaussian kernel (Vincent, 2011). Accordingly, the diffusion-equivalent
Denoising Diffusion Probabilistic Model (DDPM, Ho et al., 2020) initiates by
independently sampling $x\sim p_{\star}$ and $z\sim p_{z}$. However, recent
work has shown the benefits of choosing more adapted couplings in flow
matching then in consistency models, thereby going beyond the original
inspiration from diffusion models.

#### Beyond Independent Coupling (IC).

The reliance on IC in consistency and flow models is increasingly recognized
as a limiting factor. Recent advancements (Liu et al., 2023; Pooladian et al.,
2023) suggest that improved coupling mechanisms could enhance both training
efficiency and the quality of generated samples in flow matching. By reducing
the variance in gradient estimation, enhanced coupling can accelerate
training. Additionally, improved coupling could decrease transport costs and
straighten trajectories, yielding better-quality samples.

#### The OT Approach.

Prompted by these findings, Pooladian et al. (2023) have explored OT methods
to devise batch couplings within the framework of flow matching models. They
show that deterministic and non-crossing paths enabled by OT with infinite
batch size lowers the variance of gradient estimators. Experimentally, they
assess the efficacy of OT solvers, such as Hungarian matching and Sinkhorn
algorithms, in coupling batches of noise and data points. Dou et al. (2024)
have adopted this approach in consistency models, underscoring the utility of
batch OT in boosting model performance. However, a significant challenge
remains in the generative modeling of images, where the coupling’s
effectiveness is limited by the meaningfulness of distances between noise and
natural images, and by the batch size.

Another line of works using OT tools with score-based models relies on the
Schrodinger Bridge formulation (De Bortoli et al., 2021; Shi et al., 2024;
Korotin et al., 2024). However, it has mostly proven benefits on transfer
tasks rather than standard generative tasks.

#### Re-using ODE Couplings.

An alternative strategy, termed ReFlow (Liu et al., 2023), leverages existing
couplings provided by the ODE in a flow framework. Here, the ODE is defined as
${\operatorname{d\\!}x}=v(x_{t},t)\operatorname{d\\!}t$ with intermediate
points calculated as $x_{t}=tx+(1-t)z$ and $t\in[0,1]$. Once a neural network
$v_{\theta}$ is trained using these ICs, solving the backward ODE defines a
deterministic function mapping each sample noise to a sample from the
approximated data distribution. Notably, Liu et al. (2023) show that this new
coupling can reduce transport costs and straighten trajectories, and can be
used iteratively for training models from straighter and straighter
trajectories. The authors note however that using this procedure with a PF-ODE
solver would not guarantee decreasing transport costs and straightening
trajectories. Nonetheless, in this iterative procedure, errors can accumulate
due to approximation errors from $v_{\theta}$ and the discretization in the
ODE solver.

##  4 Consistency models with Generator-Induced Coupling (GC)

In this section we introduce our method, named GC, leveraging the generator
outputs during training. It consists in using the consistency model itself to
re-define the standard independent coupling (IC). We run a series of
experiments motivating the use and giving intuition of GC. Notably, we show
that the resulting trajectories are closer to the ideal PF-ODE updates than
with standard IC.

###  4.1 Generator-Induced Coupling (GC) Training

The solution proposed in this work, illustrated in Figure 1, consists in
leveraging the consistency model itself to build a novel type of coupling. The
idea is to leverage the properties and the accumulated knowledge from the
consistency model itself, $f_{\theta}$, to construct pairs of points. To do
so, the first step is to sample an intermediate point, which is done as usual
with $x\sim p_{\star}$, $z\sim p_{z}$ and an IC between the two distributions,
and then predicting its endpoint $\hat{x}$ via the consistency model:

$\displaystyle(x,z)\sim q_{\mathrm{I}},$ $\displaystyle x_{i}=x+\sigma_{i}z,$
$\displaystyle\hat{x}=\mathrm{sg}(f_{\theta}(x_{i},\sigma_{i})).$ (7)

and this $\hat{x}$ is coupled to $z$, thereby defining our _generator-induced
coupling_ $q$, to construct the pair of points
$(\tilde{x}_{i},\tilde{x}_{{i+1}})$:

$\displaystyle(\hat{x},z)\sim q,$
$\displaystyle\tilde{x}_{i}=\hat{x}+\sigma_{i}z,$
$\displaystyle\tilde{x}_{{i+1}}=\hat{x}+\sigma_{{i+1}}z.$ (8)

The loss function is defined on this generator-induced pair of points:

$L_{\text{g-consistency}}=\mathbb{E}_{(\hat{x},z)\sim
q,\tilde{x}_{i}=\hat{x}+\sigma_{i}z,\tilde{x}_{i+1}=\hat{x}+\sigma_{i+1}z}\Big{[}\mathcal{D}\Big{(}\mathrm{sg}(f_{\theta}(\tilde{x}_{i},\sigma_{i})),f_{\theta}(\tilde{x}_{i+1},\sigma_{i+1})\Big{)}\Big{]}.$
(9)

The overall procedure is illustrated in Figure 1 and presented in Algorithm 1.

#### Generator-induced trajectories satisfy the boundary conditions of
diffusion processes.

Let us note two following important properties of the distribution
$p_{\tilde{x}_{i}}$ of $\tilde{x}_{i}$: $p_{\tilde{x}_{0}}=p_{0}\approx
p_{\star}$ and $p_{\tilde{x}_{N}}=p_{\sigma_{N}}\approx p_{z}$. The first
property is achieved thanks to the boundary condition of the consistency model
(c.f. Section 2.1), and the second property by construction of the GC $q$
preserving the marginal noise distribution.

Input: Randomly initialized consistency model $f_{\theta}$, number of
timesteps $k$, noise schedule $\sigma_{i}$, loss weighting $\lambda(\cdot)$,
learning rate $\eta$, distance function $\mathcal{D}$, noise distribution
$p_{z}$.

Output: Trained consistency model $f_{\theta}$.

1 while  _not converged_ do

$x\sim p_{\star}$ ;

// set of real data points

$z\sim p_{z}$ ;

// set of noise vectors

$i\sim\text{multinomial}\big{(}p(\sigma_{0}),\ldots,p(\sigma_{N})\big{)}$ ;

// sampling timesteps

$x_{{i}}\leftarrow x+\sigma_{i}z$ ;

// computation of intermediate points

2

$\hat{x}\leftarrow\mathrm{sg}\big{(}f_{\theta}(x_{i},\sigma_{i})\big{)}$ ;

// endpoint prediction from the consistency model

3

$\tilde{x}_{i}\leftarrow\hat{x}+\sigma_{i}z$,
$\tilde{x}_{{i+1}}\leftarrow\hat{x}+\sigma_{{i+1}}z$ ;

// generator-induced intermediate points

4

$L(\theta)=\lambda(\sigma_{i})\mathcal{D}\big{(}\mathrm{sg}(f_{\theta}(\tilde{x}_{i},\sigma_{i})),f_{\theta}(\tilde{x}_{{i+1}},\sigma_{i+1})\big{)}$
;

// compute consistency loss

$\theta\leftarrow\theta-\eta\nabla_{\theta}L(\theta)$ ;

// back-propagate consistency loss

5

Algorithm 1 Training of consistency models with generator-induced
trajectories.

###  4.2 Experimental Insights into Model Properties

In this section, we propose to experimentally validate the soundness of our
proposed coupling procedure by evaluating key quantities. Sections 2.2 and 3
expose three key properties that could play a role in accelerating the
training and improve the performances of consistency models: _(i)_ a lower
variance of the gradient estimator; _(ii)_ a lower data to noise coupling
transport cost; and _(iii)_ pairs $(\tilde{x}_{i},\tilde{x}_{i+1})$ closer to
the PF-ODE trajectory, as in the distillation setting, than $(x_{i},x_{i+1})$
in IC.

#### Experimental settings.

To observe whether our proposed method verify these properties in practice, we
run experiments on two synthetic settings where the data distribution
$p_{\star}$ is a mixture of 2 Gaussians with equal weight, and the noise
distribution $p_{z}$ is either a single Gaussian or a mixture of 2 Gaussians
with equal weight. The goal of the first (resp. second) setting is to map a
1-mode (resp. 2-mode) noise distribution to a 2-mode data distribution; we
denote this setting 1m-2m (resp. 2m-2m). The 2m-2m setting, inspired by Liu et
al. (2023), facilitates the visualization of intersecting trajectories. We
then compare our GC training to the standard IC training.

####  _(i)_ Lower variance of the gradient.

In this experiment, we propose to measure how the variance of the estimator of
the gradient of the GC training compares to the one of the standard IC
training. As highlighted in Section 3 a lower variance could positively impact
the training and performances. We then run a full training, in both 1m-2m and
2m-2m settings for both GC and IC trainings, and log the variance for each
batch. As shown in Figure 2, the GC procedure lowers the variance of the
gradient’s estimator in both settings. Note that, particularly in the 2m-2m
setting, using an Exponential Moving Average (EMA) of the parameters of the
network on top of our method smoothes the variance specifically at the
beginning of the training (around iteration $200$) and lowers it as well.

Figure 2: Evolution of the variance of the estimator of the gradient during
training of both 1m-2m and 2m-2m synthetic settings. Generator-induced
Coupling (GC) training reduces the variance compared to IC training used in
the original consistency training algorithm.

(a) Trajectories. (b) Transport cost.

Figure 3:  GC and IC trainings on 2m-2m. (a) GC samples (

▲

) are closer to the shortest paths between noise and data distributions than
IC (★). (b) Generator-induced trajectories have lower transport cost than
standard trajectories. In both cases, transportation cost has two modes.

####  _(ii)_ Lower transport costs.

As stressed in Section 3, a line of works has brought evidence that reducing
the data-noise coupling transport cost could fasten the training and help
produce better samples. We compare the quadratic transport costs of mapping an
intermediate point to its associated true or synthetic data, formally
$\mathbb{E}[\|{x}-x_{t}\|_{2}^{2}]$ and
$\mathbb{E}[|\hat{x}-\tilde{x}_{t}|_{2}^{2}]$. The consistency model used for
generating $\tilde{x}_{t}$ has been trained for $2000$ steps and has not
reached convergence yet. We run this experiment on both 1m-2m and 2m-2m
synthetic settings. Results of the 2m-2m setting are depicted in Figure 3; the
same conclusions hold for the 1m-2m settings (see Appendix Figure 8). We
observe that
$\mathbb{E}[\|{x}-x_{t}\|_{2}^{2}]\geq\mathbb{E}[|\hat{x}-\tilde{x}_{t}\|_{2}^{2}]$.
Additionally, we observe that GC alters the marginal probabilities, such that
$p_{\tilde{x}_{i}}\neq p_{\sigma_{i}}$ for $i\in[1,\ldots,N-1]$. Notably,
after intersections, the trajectories tend to be attracted to paths between
the closest modes. This point is confirmed on the right part of the plot where
the expected cost is far lower for GC and the two modes in the cost tend to
disappear for larger timesteps (denoting that intermediate points
$\tilde{x}_{t}$ are mapped to nearest mode).

####  _(iii)_ Closer to the PF-ODE trajectories.

As discussed in Section 2.2, the goal of consistency models enforcing
consistency on pairs of points that belong to a trajectory of the PF-ODE, but
pairs of points $(x_{i},x_{i+1})$ built on IC do not satisfy this property. We
hypothesize that our $(\tilde{x}_{i},\tilde{x}_{i+1})$ built on GC are closer
to actual PF-ODE trajectories. To do so, we first learn a score-based model
$s_{\Phi}$ in both the 1m-2m synthetic setting and on the CIFAR-10
dataset111For this experiment the 2m-2m setting would require a bridge
process. with an EDM setting. Given $s_{\Phi}$ we can compute a PF-ODE update
from any point using Equation 5. We then train consistency models with either
standard IC training or GC training. We define true PF-ODE updates
$x^{\Phi}_{i}=x_{i+1}-(\sigma_{i}-\sigma_{i+1})\sigma_{i+1}s_{\Phi}(x_{i+1},\sigma_{i+1})$
and
$\tilde{x}^{\Phi}_{i}=\tilde{x}_{i+1}-(\sigma_{i}-\sigma_{i+1})\sigma_{i+1}s_{\Phi}(\tilde{x}_{i+1},\sigma_{i+1})$.
Then, we compare their distance with the training pairs, resp.
$\mathbb{E}[\|x_{i}-x^{\Phi}_{i}\|_{2}]$ and
$\mathbb{E}[\|\tilde{x}_{i}-\tilde{x}^{\Phi}_{i}\|_{2}]$. The smaller this
quantity, the closer the update to the ideal update obtained from the score-
based model. As shown in Figure 4, GC pairs are consistently closer to actual
PF-ODE trajectories than IC ones.

Figure 4: Distance to score-based updates of standard IC pairs
$(x_{i},x_{i+1})$ and GC pairs $(\tilde{x}_{i},\tilde{x}_{i+1})$ during
training. GC pairs are closer to the ideal updates than IC ones.

##  5 Experiments on Unconditional Image Generation

In the previous section we validate experimentally several key properties and
visualize the impact of our proposed GC training on two synthetic settings. In
this section, we experiment GC training to learn consistency models on
unconditional image generation tasks without a pre-trained score model.

###  5.1 Experimental Setting

We apply our approach to three image datasets used in an unconditional
setting: CIFAR-10 (Krizhevsky et al., ), $64\times 64$ CelebA (Liu et al.,
2015), and $32\times 32$ 1k-ImageNet (Deng et al., 2009). We compare our model
with standard IC consistency training setting with IC and denote this baseline
IC. The baseline IC model is built on the training principles and techniques
outlined by Song and Dhariwal (2024). Experiments were conducted on NVIDIA
A100 40GB GPUs. More details can be found in the Appendix and the codebase is
available at  https://github.com/thibautissenhuth/consistency_GC.

To evaluate the contribution of GC w.r.t. the baselines IC and batch-OT, we
propose to select a common base model and optimization strategy based on IC
that would be used for all models, based on its performances on CIFAR-10 and
common practices.

Table 1:  FID on CIFAR-10.  Forward process | Optimizer | FID
---|---|---
EDM | Adam | $13.0$
EDM | Lion | $8.8$
Bridge | Lion | $\mathbf{8.0}$

#### Selecting base components.

We start with the standard EDM forward process (Karras et al., 2022; Song et
al., 2023; Song and Dhariwal, 2024) defined as $x_{i}=x+\sigma_{i}z$, and
compare Adam (Kingma and Ba, 2015) to Lion (Chen et al., 2024) with a similar
learning rate of $1\mathrm{e}^{-4}$. We train the IC model with both
optimizers and observe a significant improvement in FID while using Lion. We
then compare EDM and bridge forward processes with Lion. We define the bridge
process as $x_{i}=(1-\alpha_{i})x+\alpha_{i}z$, with
$\alpha_{i}=\frac{\sigma_{i}}{\sigma_{i}+1}$ and $\sigma_{i}$ taken from the
EDM process. In this way, we conserve a similar signal-to-noise ratio at given
timestep $i$. Results are presented in LABEL:tab:fid_base_model. We observe a
superior performance of the bridge process with Lion, and we thus stick to
these two components for the following experiments.

#### EMA generator-induced trajectories.

Another key component of our based model is the EMA of the network parameters
for both evaluation and GC-training. Indeed, the following question arises:
while training, should GC be computed with an EMA model? Like diffusion and
score-based models, previous works (Song and Ermon, 2020) observed that at
inference EMA models have steadier and better performance. We thus use EMA for
reporting the results of both IC and batch-OT. Additionally, and specifically
to GC, we gain some insight from previous section experiments that using EMA
during training might reduces the variance of the gradient estimator (see
Figure 2). We further report, in Figure 5(b), an ablation study in our final
setting which supports the superiority of the EMA.

###  5.2 Results

In this section we report the results of our proposed method GC and expose our
thought process towards the final GC training procedure. Note that we do not
achieve the performance of Song and Dhariwal (2024) on CIFAR-10 and
1k-ImageNet due to using smaller models, a smaller batch size ($512$ vs $1024$
on CIFAR-10, $512$ vs $4096$ on ImageNet), and fewer training steps
($100\mathrm{k}$ vs $400\mathrm{k}$).

#### Timestep scheduling.

First, we learn a consistency model using the exact same training procedure as
described in Section 4 on CIFAR-10. We compare the performance of GC w.r.t. IC
during training using the exponentially increasing number of timesteps from
Song and Dhariwal (2024). We report in Figure 5(a) the FID of one-step
generated images. We observe that, at the beginning of the training, GC allows
for a faster convergence speed than IC. Unfortunately, around the
$30\mathrm{k}$-th iteration, GC suffers from degraded performance. We
hypothesize that the timestep scheduling might not be optimal for training
consistency models with GC and thus train different GC models with a fixed
number of timesteps. Results are reported in Figure 5(a). Interestingly we
observe that, with no scheduling, GC offers a faster convergence without
performance degradation. Using $20$ timesteps, we match the performance of the
best GC results obtained with standard scheduling.

(a) Timestep scheduling comparison.

(b) EMA ablation study.

Figure 5: (a) Timestep scheduling comparison for consistency models trained
with generator-induced trajectories on CIFAR 10. (b) EMA ablation study in GC
training for $\mu=0.5$.

#### Mixing procedure.

Note that, even though the GC procedure create new pairs
$(\tilde{x}_{i},\tilde{x}_{i+1})$, it still needs to apply the consistency
model to $x_{i}$ drawn from IC. Training only on GC can then induce a
distribution shift, as illustrated in Figure 3. This can be the reason why we
previously observed a degraded performance reported in Figure 5(a). We thus
propose a simple yet efficient way to reduce this distribution shift by mixing
IC and GC pairs during training, adding only one hyperparameter. We define a
mixing factor $\mu$ used to mix standard trajectories with generator-induced
trajectories. At each training step, training pairs are drawn from GC with
probability $\mu$, while the remaining pairs are computed from standard IC. We
denote GC ($\mu=\cdot$) the resulting mixing procedure. Thus, GC ($\mu=0$)
corresponds to the standard IC procedure and GC ($\mu=1$) corresponds to the
procedure introduced in Section 4. We run the mixing procedure on the three
image dataset and include batch OT (Pooladian et al., 2023; Dou et al., 2024)
as a supplementary baseline.

As shown in Figure 6, we observe an interesting interpolation phenomenon
between $\mu=0$ to $\mu=1$. At $\mu=0$, we recover the steady FID improvement
of IC training. When increasing $\mu$, the convergence of the generative model
accelerates. However, when reaching $\mu=1$, the FID improvement rate is the
fastest at the beginning of the training, but then suffers from performance
degradation. For $0.3\leq\mu\leq 0.7$, we observe a sweet spot where the
convergence is faster than IC and, for CIFAR-10 and CelebA, the final FID
improves compared to both IC and batch-OT baseline models. We show in Figure 7
samples from CelebA $64\times 64$. GC seems to produce sharper and more
detailed images specifically when looking at eyes, mouth, and hairs of
generated faces.

Figure 6:  Performance of consistency models using batch-OT or mixed GC
($\mu=\cdot$) for different mixing factors $\mu$. GC benefits from an
increased convergence rate over standard IC and achieves better performance
than both IC and batch-OT on CIFAR-10 and CelebA for $\mu$ equal to $0.5$ and
$0.7$.

(a) Trained with IC $=$ GC ($\mu=0$).

(b) Trained with GC ($\mu=0.5$).

Figure 7:  Uncurated samples from consistency models trained on CelebA 64x64
for fixed noise vectors. Note that models trained with generator-induced
trajectories tend to generate sharper images.

#### Wall-clock Training Time.

As illustrated in Figure 6, our method converges faster than IC in terms of
the number of required training steps. However, each training step is more
time-consuming, as it necessitates a forward evaluation of the consistency
model without gradient computation. Regarding wall-clock training time on
CIFAR-10, $100\mathrm{k}$ training steps under standard conditions require
approximately $20$ hours on an A100 GPU. In contrast, $100\mathrm{k}$ training
steps employing GC extend to about $25$ hours. Importantly, despite the
increased time per iteration, the hybrid model achieves the minimum Fréchet
Inception Distance (FID) sooner than the consistency model, both on CelebA and
CIFAR-10, when considering total wall-clock training time and the number of
iterations needed for reaching the minimum.

##  6 Conclusion

In this paper, we introduce a generator-induced coupling and new trajectories
for training consistency models without a pre-trained score model. We
experimentally give intuition on the interest of those trajectories. Most
notably, they are closer to ideal trajectories from a pre-trained score update
than standard ones. This results in a faster convergence in terms of both the
number of training steps and wall-clock training time. Our approach paves the
way for a new type of coupling induced by the generator itself. Interestingly,
our method offers a fresh viewpoint on this problem, which is usually tackled
with traditional OT tools.

#### Limitations.

Training consistency models with generator-induced trajectories leads to
intriguing behaviors that we still do not fully understand. For example, why
do they exhibit different behavior than standard consistency models regarding
the scheduling of timesteps during training? This is surprising since Song et
al. (2023) showed that increasing the number of timesteps leads to a decreased
approximation error of the consistency model. Future work should carefully re-
examine hyperparameter choices for consistency models trained with generator-
induced trajectories, especially for scheduling. Another technical limitations
of our work is, at this time, the lack of theoretical support on the benefits
of generator-induced trajectories. The advantages they bring are evaluated
experimentally on synthetic settings and computer vision datasets, but then
limited to these scopes.

## References

  * Chen et al. (2024) Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al.  Symbolic discovery of optimization algorithms.  _Advances in Neural Information Processing Systems_ , 2024.
  * De Bortoli et al. (2021) Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet.  Diffusion Schrödinger bridge with applications to score-based generative modeling.  _Advances in Neural Information Processing Systems_ , 2021.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2009.
  * Dou et al. (2024) Hongkun Dou, Junzhe Lu, Jinyang Du, Chengwei Fu, Wen Yao, Xiao qian Chen, and Yue Deng.  A unified framework for consistency generative modeling, 2024.  URL https://openreview.net/forum?id=Qfqb8ueIdy.
  * Fallis (2020) Don Fallis.  The epistemic threat of deepfakes.  _Philosophy & Technology_, 2020.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33, 2020.
  * Karras et al. (2022) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35, 2022.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Kingma and Gao (2024) Diederik Kingma and Ruiqi Gao.  Understanding diffusion objectives as the elbo with simple data augmentation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Korotin et al. (2024) Alexander Korotin, Nikita Gushchin, and Evgeny Burnaev.  Light Schrödinger bridge.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * (11) Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  CIFAR-10 (Canadian Institute for Advanced Research).  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * Lipman et al. (2023) Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le.  Flow matching for generative modeling.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Liu et al. (2023) Xingchao Liu, Chengyue Gong, and qiang liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2015.
  * Meng et al. (2023) Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems_ , volume 32. 2019.
  * Pooladian et al. (2023) Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky TQ Chen.  Multisample flow matching: Straightening flows with minibatch couplings.  In _International Conference on Machine Learning_ , 2023.
  * Salimans and Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2022.
  * Sauer et al. (2023) Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation.  _arXiv preprint arXiv:2311.17042_ , 2023.
  * Shi et al. (2024) Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet.  Diffusion Schrödinger bridge matching.  _Advances in Neural Information Processing Systems_ , 2024.
  * Song and Dhariwal (2024) Yang Song and Prafulla Dhariwal.  Improved techniques for training consistency models.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Song and Ermon (2019) Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  In Hanna Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Roman Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32, pages 11918–11930. Curran Associates, Inc., 2019.
  * Song and Ermon (2020) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  _Advances in neural information processing systems_ , 33, 2020.
  * Song et al. (2021) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2021.
  * Song et al. (2023) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  In _Proceedings of the 40th International Conference on Machine Learning_ , 2023.
  * Vincent (2011) Pascal Vincent.  A connection between score matching and denoising autoencoders.  _Neural computation_ , 23(7), 2011.

##  Appendix A Broader Impacts

If used in large-scale generative models, notably in text-to-image models,
this work may increase potential negative impacts of deep generative models
such as deepfakes (Fallis, 2020).

##  Appendix B Experimental Details

First, let us recall that the code is available at
https://github.com/thibautissenhuth/consistency_GC. It is based on the PyTorch
library (Paszke et al., 2019),

#### Scheduling functions and hyperparameters from Song and Dhariwal (2024).

The training of consistency models heavily rely on several scheduling
functions. First, there is a noise schedule $\\{\sigma_{i}\\}_{i=0}^{N}$ which
is chosen as in Karras et al. (2022). Precisely,
$\sigma_{i}=\big{(}{\sigma_{\text{0}}}^{\frac{1}{\rho}}+\frac{i}{N}({\sigma_{\text{N}}}^{\frac{1}{\rho}}-{\sigma_{\text{0}}}^{\frac{1}{\rho}})\big{)}^{\rho}$
with $\rho=7$. Second, there is a weighting function that affects the training
loss. It is chosen as $\lambda(\sigma_{i})=\frac{1}{\sigma_{i+1}-\sigma_{i}}$.
Combined with the choice of noise schedule, it emphasizes to be consistent on
timesteps with low noise. Then, Song et al. (2023) proposed to progressively
increase the number of timesteps $N$ during training. In their most recent
work, they argue that a good choice of dicretization schedule is an
exponential one, which gives
$N(k)=\text{min}(s_{0}2^{\lfloor\frac{k}{K^{\prime}}\rfloor},s_{1})+1$ where
$K^{\prime}=\lfloor\frac{K}{\log_{2}[s_{1}/s_{0}]+1}\rfloor$, $K$ is the total
number of training steps, $k$ is the current training step, $s_{0}$
(respectively $s_{1}$) the initial (respectively final) number of timesteps.
Finally, Song and Dhariwal (2024) propose a discrete probability distribution
on the timesteps which mimics the continuous probability distribution
recommended in the continuous training of score-based models by Karras et al.
(2022). It is defined as
$p(\sigma_{i})\propto\text{erf}(\frac{\log(\sigma_{i+1})-P_{\text{mean}}}{\sqrt{2}P_{\text{std}}})-\text{erf}(\frac{\log(\sigma_{i})-P_{\text{mean}}}{\sqrt{2}P_{\text{std}}})$.

In practice, Song and Dhariwal (2024) recommend using: $s_{0}=10$,
$s_{1}=1280$, $\rho=7$, $P_{\text{mean}}=-1.1$, $P_{\text{std}}=2.0$.

#### Details on bridge process.

For the bridge process, note that we first sample $\sigma_{i}$ from
$\sigma_{i}=\big{(}{\sigma_{\text{0}}}^{\frac{1}{\rho}}+\frac{i}{N}({\sigma_{\text{N}}}^{\frac{1}{\rho}}-{\sigma_{\text{0}}}^{\frac{1}{\rho}})\big{)}^{\rho}$.

On Gaussians experiment, we use $\rho=3$, $\sigma_{\text{0}}=0.001$ and
$\sigma_{\text{N}}=1$. Then, we compute $\alpha_{i}=\sigma_{i}$ and define
intermediate points such as $x_{i}=\alpha_{i}x+(1-\alpha_{i})z$.

On image experiments, we use $\rho=7$, $\sigma_{\text{0}}=0.001$ and
$\sigma_{\text{N}}=80$. Then, we compute
$\alpha_{i}=\frac{\sigma_{i}}{\sigma_{i}+1}$ and define intermediate points
such as $x_{i}=\alpha_{i}x+(1-\alpha_{i})z$.

Details on neural networks architectures. On Gaussians, we use simple MLPs
with GELU activation functions. On image datasets, we use the NCSN++
architecture (Song et al., 2021) and follow the implementation from
https://github.com/NVlabs/edm.

Table 2: Hyperparameters for the Gaussians experiments. Hyperparameter | Value
---|---
number of samples | 10000
batch size | 512
training steps | 10000
learning rate | 0.00005
$s_{0}$ | 30
$s_{1}$ | 30
$\rho$ | 3
$\sigma_{0}$ | 0.001
$\sigma_{1}$ | 1
model | MLP
depth | 4
hidden dim | 256
Table 3: Hyperparameters for CIFAR-10. Arrays indicate quantities per resolution of the UNet model. Hyperparameter | Value
---|---
batch size | 512
image resolution | 32
training steps | 100000
learning rate | 0.0001
optimizer | lion
$s_{0}$ | 10
$s_{1}$ | 1280
$\rho$ | 7
$\sigma_{0}$ | 0.001
$\sigma_{1}$ | 80
network architecture | SongUNet
| (from Karras et al. (2022) implementation)
model channels | 128
dropout | 0.3
num blocks | 3
embedding type | positional
channel multiplicative factor | [1,2,2]
attn resolutions | $\emptyset$
Table 4: Hyperparameters for CelebA. Arrays indicate quantities per resolution of the UNet model. All models are trained with two learning rates and the best is selected.  Hyperparameter | Value
---|---
batch size | 128
image resolution | 64
training steps | 150000
learning rate | {0.0001, 0.00005}
optimizer | lion
$s_{0}$ | 10
$s_{1}$ | 1280
$\rho$ | 7
$\sigma_{0}$ | 0.001
$\sigma_{1}$ | 80
network architecture | SongUNet
| (from Karras et al. (2022) implementation)
model channels | 128
dropout | [0.05,0.05,0.1,0.2]
num blocks | [2,3,4,5]
embedding type | positional
channel multiplicative factor | [1,2,2,2]
attn resolutions | $\emptyset$
Table 5: Hyperparameters for ImageNet. Arrays indicate quantities per resolution of the UNet model. All models are trained with two learning rates and the best is selected.  Hyperparameter | Value
---|---
batch size | 512
image resolution | 32
training steps | 150000
learning rate | {0.0001, 0.00005}
optimizer | lion
$s_{0}$ | 10
$s_{1}$ | 1280
$\rho$ | 7
$\sigma_{0}$ | 0.001
$\sigma_{1}$ | 80
network architecture | SongUNet
| (from Karras et al. (2022) implementation)
model channels | 128
dropout | [0.1,0.3,0.3]
num blocks | [4,5,7]
embedding type | positional
channel mult | [1,1,2]
attn resolutions | 16

Details on computational ressources As mentioned in the paper, the image
dataset experiments have been conducted on some NVIDIA A100 40GB GPUs. The
Gaussians experiments have also been computed on GPUs (V100 or A100) and run
in few minutes.

##  Appendix C Additional Experimental Results on Gaussians

We show for the Gaussian experiment, transportation cost of couplings in the
1m-2m setting in Figure 8.

Figure 8:  GC and IC trainings on 1m-2m. (a) GC samples are closer to the
shortest paths between noise and data distributions than IC. (b) Generator-
induced trajectories have lower transport cost than standard trajectories.


11institutetext: College of Computer Science & Information Systems, Najran
University, Najran, Saudi Arabia

# Tasks Decomposition Approaches in Crowdsourcing Software Development

Abdullah Khanfor

###### Abstract

A main characteristic of crowdsourcing software development (CSD) is the
complexity of tasks and skills required by workers to achieve successful
software crowdsourcing. The tasks proposed to the crowd in CSD are checked to
ensure they are manageable and achievable. In general, individual tasks come
from general goal-oriented projects. There are practices for breaking down
software projects into manageable tasks, known as task decomposition. This
study identified task decomposition techniques in software engineering,
particularly in the context of CSD. Then, we defined the experienced
developers who lead the requester in decomposing the project, preparing tasks,
and reviewing submissions. This study explored and addressed decomposition
approaches in CSD. Next, we selected projects in TopCoder to identify the task
decomposition process in the CSD context. Finally, we concluded with future
research directions for investigating decomposition approaches and their
effects in the CSD context to ensure successful crowdsourced software
projects.

###### Keywords:

Crowdsourcing Software Development Software Decomposition Tasks Decomposition
Crowdsourced Software Project.

††This paper is accepted for publication in the 25th International Conference
on Human-Computer Interaction, Jul. 2023.
The Conference Proceedings will be published by © 2023 Springer. Personal use
of this material is permitted. Permission from Springer must be obtained for
all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  1 Introduction

The power and potential of crowdsourcing have been demonstrated in multiple
fields using various statistical tools. Moreover, affordable, high-quality
products and solutions should be created from diverse, experienced backgrounds
to solve problems and fulfill the owner’s needs. “Crowdsourcing is the act of
taking a job traditionally performed by a designated agent (usually an
employee) and outsourcing it to an undefined, generally large group of people
in the form of an open call” [6]. CSD is derived from crowdsourcing ideas
through the proposal of software tasks such as developing, testing, and bug
fixing in an open-call format [18, 14]. In some circumstances, CSD is more
appealing to developers and companies. CSD reduces costs and improves
deliverable quality through reducing defect rates [11].

On CSD platforms, the requester can ask for assistance in preparing and
managing tasks from a pool of experienced workers. These experienced workers,
known as copilots in TopCoder, improve the task description, reward, and
communication with other workers to ensure the success of the task or project.
Workers register for tasks to submit artifacts. Moreover, in TopCoder,
submissions from the crowd are assigned scores out of 100 points. If there is
no submission to the task, it is labeled as a failure, also known as task
starvation [8]. If there are submissions with scores less than 75, the tasks
are marked as a failure, and the task is canceled because of the low quality
of the submissions. If the task submissions have a score of 75 or above, the
top submissions are selected as winners, and a reward is given to the top
submitters. Moreover, the copilot receives a separate award for guiding the
requester and crowd to successfully completing the task. The CSD process using
copilots is illustrated in Fig. 1.

Figure 1: Copilots involvement in the general CSD workflow.

Various studies have examined the role of copilots in the CSD context. A
significant role of copilots in CSD projects is task decomposition, which
breaks down the project into a set of microtasks to ensure deliverable
software quality. However, there are some conditions and trade-offs in
choosing appropriate strategies for decomposing a project [20, 12, 17]. On the
one hand, efficient task decomposition for a software project requires
detailed specifications to capture the project’s requirements. On the other
hand, it leaves room for innovation from the crowd [11]. Hence, The detailed
decomposition can increase overhead communication, and coordination with
workers [13]. Moreover, in addition to the accurate description of the task,
decomposed tasks should have limited interdependences to ensure success [21].
Therefore, methods for efficient task decomposition in the CSD context should
be identified and examined. To this end, we identify approaches used in task
decomposition to build a systematic approach that accelerates the CSD process
and its adoption for software projects.

##  2 Background

This section examines software decomposition, task decomposition practices
used in software engineering, and various decomposition terminologies, such as
cohesion, coupling, and horizontal and vertical decomposition. Next, we
highlight the main tasks of the copilot, such as task preparation and
decomposition. The following is a background discussion of various software-
task decomposition concepts and the role of the copilot in CSD.

###  2.1 Software Decomposition

The goal of software engineering decomposition is to reduce complexity [1].
The design of complex systems involves three critical structures [15]. 1) The
module structure assigns an individual programmer or team to closely related
programs, where the module is a group of decomposed programs. 2) For the users
of the structure, the components are part of the modules. 3) The process
structure decomposes runtime activities, known as a process. There are no
direct relationships between the modules and process, which is challenging to
identify. During task decomposition, the following criteria must be considered
when decomposing tasks. First, the duplication of tasks should be avoided.
Second, the dependency of project tasks should be identified; for example, if
Task 2 depends on the completion of Task 1, this dependency must be maintained
during task design. Finally, the generality concept in software development is
when we can use other components in various situations without changes, which
can be driven to task generality. Therefore, we can reuse other similar
successful tasks for different projects in the CSD context.

####  2.1.1 Cohesion & Coupling

The components of task decomposition must be cohesive and coupled. In task
decomposition, a high cohesion between different tasks can indicate their
similarity and vice versa. The increased coupling means a slight change in one
subsystem can significantly impact other subsystems. The same concept can be
applied to task decomposition. Thus, tasks with high coupling and dependency
require adjustments to the other tasks.

####  2.1.2 Horizontal & Vertical Decomposition

Decomposition in the Agile methodology can be divided into two main types: (1)
horizontal and (2) vertical split [5]. Horizontal split is the traditional
approach in which each significant feature in the software is decomposed based
on architectural layers. Therefore, the role of each layer may be assigned to
a different individual or team. A significant benefit of this approach is the
building of expertise among responsible specialists based on the assigned
layer. Vertical decomposition involves decomposing a prominent feature that
overlaps a different architectural layer. The breakdown satisfies the
requirement that the development of each component, including construction and
integration into the testing component, must take a few days. Green [5] stated
that the benefits of adopting the vertical decomposition approach could rely
on a simple understanding, more accurate estimation, increasing the feedback
loop, and noting the progress of a software project. Fig. 2 shows the vertical
and horizontal slices of the client-server architecture.

Figure 2: Client-server Architecture and Vertical Scales Demonstration [5].

####  2.1.3 Decomposition Levels

Each task should be labeled according to Cockburn’s altitude notation, Fig 3,
where tasks with too abstract goals are placed at the sky level, and software
goals with no precise ending are placed at the kite level. At the kite level,
CSD task decomposition is more intense. The greater detail of the functional
and sea levels allows developers to complete the tasks in these levels in a
few sittings. At the sub-functional level, each task may not individually
indicate the goal of the software; however, multiple sub-functions can
demonstrate the goal. Therefore, this level is the optimal level for tasks
that can lead to higher parallelism and reduce the dependability of tasks.
Finally, the deep-sea level or shell level should be more detailed and focus
on the goal of the task, with less visibility of the project goals. In Fig 3,
illustration of different levels of requirements description.

Figure 3: Illustration of different levels of requirements description.

###  2.2 Copilot roles in CSD

Saengkhattiya et al. [16] studied how quality is handled by crowdsourced
companies. One such technique involves conducting interviews and
questionnaires. They found that copilots generally act as “middlemen between
the task requester and workers. The copiloting process includes discussions
regarding various aspects of CSD, such as project planning, pricing,
communicating, and writing task descriptions. Thus, the copilot faces
considerable challenges that they must mitigate and resolve. The authors
noticed several runs of testing and individual assessments performed by
copilots that ensured high-quality worker submissions. Communication among
different parties is a challenge in CSD [19]. On the one hand, having one
point of contact, the copilot, with whom the project owner can communicate is
advantageous. On the other hand, this single line of communication inhibits
direct communication between the workers and the project owner, especially in
topics such as how to submit their solutions. Saengkhattiya et al. [16]
reported communication and coordination problems between customers and
copilots; these communication problems lead to copilot changes initiated by
the customer. The copilot can prevent the requester from hearing directly from
the crowd community. In most cases, utilizing copilots increases the overall
cost of the project.

A crucial role of the copilot is to prepare for task decomposition and
identify dependencies among crowdsourced software projects. Farraj [4]
proposed concepts that can be employed on a software-driven automated copilot
tool based on a predefined workflow; however, the empirical validation of the
recommended tool was limited. The primary motivation for developing the tool
was that copilots might unexpectedly leave a project or mismanage it, leading
to its failure or delays.

TopCoder defines a copilot as a member who manages the TopCoder process for a
customer to deliver the requested asset. The copilot controls the crowd’s
direction and is responsible for working with the customer at an appropriate
level of detail to convert the business submission into technical requirements
[16]. Copilots earn money by successfully completing challenges in three main
tracks: design, development, and data science. Copilots are part of the crowd
that applies for this role and are certified after achieving a set of minimum
eligibility requirements The evaluation process can be divided into two parts:
(1) pre-requisite requirements, including submitting and winning a specific
number of challenges, in addition to having a certain number of posts in the
TopCoder Forums, and (2) written exams, including writing challenge
specifications, estimating budgets and timelines, and providing strategic
feedback to customers. Applicants are evaluated based on the content of their
answers and written English skills, which are necessary for clear
communication with customers and the crowd. Furthermore, TopCoder continues to
monitor copilot performance after the copilot is certified. While working with
an actual customer, TopCoder members evaluate the copilot from many
perspectives, including customer management, community management, technical
skills, communication, and professionalism.

Figure 4: Copilot contribution in the CSD project workflow.

In Fig. 4, we summarize the roles of the copilot during the project life
cycle. In Step 1, the requester starts the project. During project initiation,
the requester may select a copilot from a pool of copilots. Usually, copilots
are matched by TopCoder (Platform) based on the utilization, availability,
skills, and experience of the copilot for the new project. TopCoder creates a
shortlist of possible copilots that satisfies previous criteria from which the
project owner can select a copilot. Next, the selected copilot begins
communicating with the project owner to clarify or discuss the project scope,
plan, and requirements, as shown in Steps 2 and 3. The copilot takes the lead
in creating tasks and the project timeline in Step 4, which decomposes
different tasks for the crowd. After posting the task to the crowd in Step 5,
the copilot communicates with workers to clarify the requirements of the
created tasks. The copilot monitors the published tasks and ensures a high
level of participation in Step 6. If required, the copilot may request the
platform to announce the tasks to the crowd. The crowd performs the tasks, and
after they submit their solutions, another set of crowd reviewers examines the
deliverables of the tasks in Steps 7 and 8. Finally, in Step 9, the copilot
reviews the submissions to ensure that the project is delivered successfully.
In some cases, the workers may appeal to the review score, and the copilot
analyzes the submissions and reviews to override the reviewer’s evaluation.

###  2.3 Decomposition in Crowdsourced Tasks

CrowdForge [9] proposed a framework to solve dependent and complex tasks using
a map-reducing approach. Turkomatic [10] studied the design of the workflow of
a task. It provides task decomposition that the requester can change. Another
example is Cascade [2], an automated object taxonomy that generates labels and
selects the best one for categorization. Therefore, it creates highly
parallelized mini-tasks that can be completed in 20 s. Despite these
frameworks, their primary concern is with the MTurk task platform, which are
short and low-reward tasks that do not require specific skills, unlike CSD
Tasks. Therefore, their conclusions can be used but are implied in the CSD
task scenario. Eickhoff and de Vries [3] found that a unique phrasing of the
task description affects outcome quality. Thus, careful task decomposition
that considers this finding will help provide a more robust framework that
aids in task decomposition. Jiang and Matsubara [7] conducted a simulation to
decompose a complex task into mini-tasks using two models. The authors
analyzed vertical and horizontal task decomposition simulation results and
found that, in general, vertical decomposition outperforms horizontal
decomposition in obtaining better quality outcomes from the crowd.

##  3 Methodology

The study has two main objectives. First, identifying whether the
decomposition approach of tasks in the historical CSD project was horizontal
or vertical. To achieve this, we examined historical CSD tasks within projects
using the technology required to accomplish the task. From there, if most of
the tasks within a CSD project use both frontend and backend programming
languages for each task, we conclude that the tasks were vertically
decomposed; if a small percentage of tasks are denoted as both, and there is a
more significant number of tasks that can be distinguished as frontend and
backend, the tasks were horizontally decomposed. Second, we determine whether
the projects selected for this study can be used to observe the effect of task
decomposition on the success rate and overall success of CSD projects.

The historical dataset was collected from the TopCoder platform from January
2014 to January 2015. It contains 4907 tasks. Of the tasks, 84.3% were marked
as complete, indicating that the code submitted by the workers was rated, and
the requester accepted it. The dataset contains the tasks and developer
information. In each task, the associated metadata describe the task,
including the task ID, title of the task, start dates for registration and
submission, the end date of submissions, task type, and task description. Each
task has several registered workers, and several submissions, and depending on
whether it is completed, it is assigned a success or failure status. TopCoder
defines a “completed” or a successful task as one with a submission with a
score of at least 75 out of 100. If a submission has a score lower than 75 or
there are zero submissions, the task is marked as “canceled.”

##  4 Results

We analyzed the types of tasks based on these technologies. We observed that
there were 75 programming languages or frameworks in the proposed tasks, for
example, PHP, iOS, and Node.JS. Of the identified technologies, 23 are used
for front-end development, whereas 61 are used in back-end development. Note
that 23 and 61 add up to more than 75 technologies, which is because some of
the technologies can be used for both frontend and backend development. This
helps us distinguish which type of decomposition is used for the tasks
proposed in TopCoder. Fig. 5 shows the percentages of frontend and backend
tasks. From the figure below, we can conclude that Project $5104$ is
horizontally decomposed. For Project $7424$, the frontend and backend tasks
represent the highest percentage of tasks in the project. Therefore, this
project was vertically decomposed.

Figure 5: Percentages of Task Type in 6 Projects and the Overall Dataset.

To examine the effect of the type of task decomposition on CSD, we used the
failure rate of each project. The failure rate of the dataset was 15.7%. Thus,
if the project has a lower failure rate than the overall failure rate, it is
assumed to be successful. For example, in Fig. 6, we examine the selected
projects, as stated, Project 5104 is horizontally decomposed and had a
successful task rate. In contrast, project 7424, which is vertically
decomposed, has a task success rate that is lower than the average overall
success rate on the platform.

Figure 6: Tasks failure rate for different projects and the overall dataset.

The award, complexity, duration, and number of workers in the task and their
influence on the task’s success can help us identify which part of the CSD
requires in-depth research [8]. The strong effect of task description on the
task outcome acts as motivation for further research. Using the right task
decomposition method can reduce the complexity of tasks by ensuring sufficient
workload and price. In Fig. 6, we observe that projects 7251, 6243, and 7424
have a higher failure rate than the overall failure rate of the tasks in the
historical dataset, where the blue dashed line represents the acceptable
failure rate of CSD projects. Meanwhile, projects 6275, 5104, and 6123 can be
considered successful projects because they exceed the failure rate threshold
(dashed line) of the referenced CSD projects.

##  5 Conclusion and Future Work

This study investigated decomposition methods for software projects. First, we
provide an overview of the role of copilots in CSD. A main task of a copilot
is decomposing software projects into a set of tasks. Therefore, the
decomposition techniques used in software engineering are reviewed. Next, we
identified the vertical and horizontal decomposition methods. Then, we studied
the decomposition of selected CSD projects. This study paves the way for
investigating the importance of decomposing tasks to ensure the success of
crowdsourced software projects. In future work, we will compare vertical and
horizontal decompositions on a large scale and their impact on the success
rate of CSD tasks.

## References

  * [1] Bruegge, B., H Dutoit, A.: Object-Oriented Software Engineering Using UML, Patterns, and Java™ Third Edition. by Pearson Education, Inc., (2010)
  * [2] Chilton, L.B., Little, G., Edge, D., Weld, D.S., Landay, J.A.: Cascade: Crowdsourcing taxonomy creation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp. 1999–2008. ACM (2013)
  * [3] Eickhoff, C., de Vries, A.P.: Increasing cheat robustness of crowdsourcing tasks. Information retrieval 16(2), 121–137 (2013)
  * [4] Farraj, B.Y.A.: Component-based crowdsourcing in software engineering
  * [5] Green, P.: Agile @ Adobe ideas, experiences, and challenges related to adobe’s journey towards agility. http://blogs.adobe.com/agile/2013/09/27/splitting-stories-into-small-vertical-slices/, accessed: 2018-02-07
  * [6] Howe, J.: The rise of crowdsourcing. Wired magazine 14(6), 1–4 (2006)
  * [7] Jiang, H., Matsubara, S.: Efficient task decomposition in crowdsourcing. In: International Conference on Principles and Practice of Multi-Agent Systems. pp. 65–73. Springer (2014)
  * [8] Khanfor, A., Yang, Y., Vesonder, G., Ruhe, G., Messinger, D.: Failure prediction in crowdsourced software development. In: 2017 24th Asia-Pacific Software Engineering Conference (APSEC). pp. 495–504. IEEE (2017)
  * [9] Kittur, A., Smus, B., Khamkar, S., Kraut, R.E.: Crowdforge: Crowdsourcing complex work. In: Proceedings of the 24th annual ACM symposium on User interface software and technology. pp. 43–52 (2011)
  * [10] Kulkarni, A.P., Can, M., Hartmann, B.: Turkomatic: automatic recursive task and workflow design for mechanical turk. In: CHI’11 Extended Abstracts on Human Factors in Computing Systems. pp. 2053–2058. ACM (2011)
  * [11] Lakhani, K.R., Garvin, D.A., Lonstein, E.: Topcoder (a): Developing software through crowdsourcing. Harvard Business School General Management Unit Case (610-032) (2010)
  * [12] Lakhani, K.R., Tushman, M.L., Baldwin, C., Grandori, A., von Hippel, E., et al.: Open innovation and organizational boundaries: The impact of task decomposition and knowledge distribution on the locus of innovation1. Tech. rep., Working Paper (2012)
  * [13] LaToza, T.D., van der Hoek, A.: Should software developers be replaced by the crowd?
  * [14] Mao, K., Yang, Y., Wang, Q., Jia, Y., Harman, M.: Developer recommendation for crowdsourced software development tasks. In: Service-Oriented System Engineering (SOSE), 2015 IEEE Symposium on. pp. 347–356. IEEE (2015)
  * [15] Parnas, D.L., Clements, P.C., Weiss, D.M.: The modular structure of complex systems. In: Proceedings of the 7th international conference on Software engineering. pp. 408–417. IEEE Press (1984)
  * [16] Saengkhattiya, M., Sevandersson, M., Vallejo, U.: Quality in crowdsourcing-how software quality is ensured in software crowdsourcing (2012)
  * [17] Saremi, R., Yang, Y., Khanfor, A.: Ant colony optimization to reduce schedule acceleration in crowdsourcing software development. In: International Conference on Human-Computer Interaction. pp. 286–300. Springer (2019)
  * [18] Shahin, M., Babar, M.A.: Improving the quality of architecture design through peer-reviews and recombination. In: Software Architecture, pp. 70–86. Springer (2015)
  * [19] Stol, K.J., Fitzgerald, B.: Two’s company, three’s a crowd: a case study of crowdsourcing software development. In: Proceedings of the 36th International Conference on Software Engineering. pp. 187–198. ACM (2014)
  * [20] Szajnfarber, Z., Vrolijk, A., Crusan, J.: Exploring the interaction between open innovation methods and system complexity. In: 4th international engineering systems symposium, Hoboken, New Jersey (2014)
  * [21] Tajedin, H., Nevo, D.: Determinants of success in crowdsourcing software development. In: Proceedings of the 2013 annual conference on Computers and people research. pp. 173–178. ACM (2013)