# Regression to the Mean’s Impact on the Synthetic Control Method: Bias and
Sensitivity Analysis

Nicholas A. Illenberger, Dylan S. Small, Pamela A. Shaw

###### Abstract

To make informed policy recommendations from observational data, we must be
able to discern true treatment effects from random noise and effects due to
confounding. Difference-in-Difference techniques which match treated units to
control units based on pre-treatment outcomes, such as the synthetic control
approach, have been presented as principled methods to account for
confounding. However, we show that use of synthetic controls or other matching
procedures can introduce regression to the mean (RTM) bias into estimates of
the average treatment effect on the treated. Through simulations, we show RTM
bias can lead to inflated type I error rates as well as decreased power in
typical policy evaluation settings. Further, we provide a novel correction for
RTM bias which can reduce bias and attain appropriate type I error rates. This
correction can be used to perform a sensitivity analysis which determines how
results may be affected by RTM. We use our proposed correction and sensitivity
analysis to reanalyze data concerning the effects of California’s Proposition
99, a large-scale tobacco control program, on statewide smoking rates.

Keywords: Regression to the mean, difference-in-difference, matching,
synthetic controls

##  1 Introduction

In observational research we must grapple with the fact that observed
differences between pre-treatment and post-treatment outcomes may not be the
result of treatment alone. Outside events, random error, and outcome trends
unrelated to treatment can add noise to our data, making it difficult to
determine the effects of an intervention. The difference-in-difference (DID)
estimator [4] attempts to provide a solution to this problem.

Given a treated and control unit with outcomes measured pre- and post-
intervention, DID is the difference in pre-treatment outcomes subtracted from
the difference in post-treatment outcomes. If we assume that the treated and
control groups would have parallel outcome trends in the absence of treatment,
then the estimator is unbiased for the average treatment effect on the treated
(ATT) even in the presence of unmeasured confounding. Ease of use and
robustness to unmeasured confounding, has made the DID popular among
epidemiologists. The effects of background checks on firearm homicide and
suicide rates [17], the development of urban green space on crime rates [7],
and policy changes on the health of mothers receiving nutritional support [15]
have all been assessed using the DID framework. However, because the estimator
is not robust to deviations from the parallel trends assumption, control units
must be selected with care.

To improve the selection of controls, Abadie et al. [1] introduced the method
of synthetic controls. Here, a new control unit is constructed using a
weighted sum of donor controls. If we weigh the donors such that they closely
resemble the treated unit in the pre-intervention period, then under standard
assumptions they should be a plausible representation of how the treated unit
would look without treatment in the post-intervention period. This method
works similarly to matching. Control units which are more similar to the
treated unit are given larger weights in the synthetic control than those
which are dissimilar. Because the synthetic control method uses information
from all control units, estimates of the ATT will tend to be less variable
than estimates obtained from one-to-one matching techniques. While matching
may improve comparability between treated and control units, recent work by
Daw et al.[9] have shown that nearest-neighbor matching can induce bias in DID
estimates due to regression to the mean (RTM). Because of the similarities
between matching and the synthetic control method, there is a need to better
understand how RTM can affect the synthetic control estimator.

In this paper, we examine the effect of RTM on estimates of the ATT coming
from the synthetic control and other matched DID methods. Through simulations,
we show that RTM can result in inflated type I error rates and, in some
settings, decreased power. Compared to other matching techniques, these
effects are exaggerated in the synthetic control estimator. We also propose a
novel sensitivity analysis, which can be used to check how robust inference
may be to the effect of RTM bias. Sensitivity and quantitative bias analyses
allow researchers to assess the potential effects of systematic error in an
experiment [19]. These approaches are common in causal inference and missing
data settings [20], and have been used to quantify the uncertainty associated
with measurement error [24]. We apply our proposed sensitivity analysis to
reanalyze data from Abadie et al.[2], estimating the effect of a large-scale
tobacco control initiative on smoking cessation rates in California.

##  2 Methods

###  2.1 Matched Difference-in-Difference

Consider a scenario in which we have two time periods, $t=0$ for the pre-
treatment period and $t=1$ for the post-treatment period. Let $Y(t)$ represent
the observed outcome at time $t$, $A$ be an indicator of treatment status, and
let $X$ be either measured or unmeasured confounders. We further define
$Y^{a}(t)$ to be the potential outcome [21] which would be observed under
treatment $A=a$ at time $t$. In this setting $Y^{1}(0)=Y^{0}(0)$ because
neither group receives treatment at time $t=0$. The DID estimator assumes the
linear model, $\mathbb{E}\left[Y^{0}(t)|X\right]=\beta X+\gamma t$, for the
expected potential outcome under no treatment at time $t$. The distribution of
$X$ will typically differ between the group truly receiving treatment and the
control group. Because of this, we expect that the potential mean under no
treatment will differ between the treatment groups.

Note that the effect of time, $\gamma$, does not depend on confounders.
Likewise the effect of confounders, $\beta$, does not depend on time. These
are jointly known as the parallel trends assumption. If both are true and if
the distribution of covariates within each group remains the same from period
to period, then the expected difference between the potential untreated
outcomes for the the treated and control units in the pre-treatment period is
equivalent to that in the post-treatment period. This is the case regardless
of the value of confounders, $X$, within these units. We define $D(t)$ to be
the expected difference between the treatment and control groups at time $t$.
Alongside these homogeneity assumptions, we also assume consistency,
$Y^{A}(t)=Y(t)$, and $Y^{0}(t)\rotatebox[origin={c}]{90.0}{$\models$}A|X$.
Consistency states that the potential outcome under the treatment received is
equivalent to the observed outcome. The second assumption, that the potential
untreated outcome is independent of treatment conditional on $X$, is likely to
hold because $X$ can contain any measured or unmeasured confounders. Under
these assumptions, we can show (see Appendix):

$\displaystyle D(t)$
$\displaystyle=\mathbb{E}\left[Y(t)|A=1\right]-\mathbb{E}\left[Y(t)|A=0\right]$
$\displaystyle=\mathbb{E}\left[Y^{1}(t)-Y^{0}(t)|A=1\right]+\beta\left(\mathbb{E}[X|A=1]-\mathbb{E}[X|A=0]\right).$

For $t=1$, the first term in this summation is the ATT, which we define as
$\theta$. Because $Y^{1}(0)=Y^{0}(0)$, it follows that $\theta$ is the
difference between $D(1)$ and $D(0)$. The DID estimator suggests using the
empirical means within treatment groups at times $t=0$ and $t=1$ to estimate
this difference. That is, $\hat{\theta}=\widehat{D}(1)-\widehat{D}(0)$, where

$\widehat{D}(t)=\frac{\sum_{i=1}^{n}Y_{i}(t)\mathbb{I}(A_{i}=1)}{\sum_{j=1}^{n}\mathbb{I}(A_{j}=1)}-\frac{\sum_{i=1}^{n}Y_{i}(t)\mathbb{I}(A_{i}=0)}{\sum_{j=1}^{n}\mathbb{I}(A_{j}=0)}.$

In practice, it may be difficult to identify units such that $\beta$ and
$\gamma$ are equivalent in the treated and control groups. Ryan et al. [22]
have shown that matching can decrease bias in some cases by improving the
comparability of units. However, as we will discuss, matching can have the
effect of introducing bias into estimates of the DID.

###  2.2 Regression to the Mean

Regression to the mean (RTM) is a statistical phenomena in which extreme
measurements of a random variable tend towards their expected value upon
repeat measurement. If not properly accounted for, RTM can lead to misleading
results. Compared with the widespread use of matched DID methods, relatively
little work has been done to examine the effects of RTM in this setting. Daw
et al. [9] have found that nearest-neighbor matching can induce bias in
estimates of the ATT when using DID techniques. However, as far as we are
aware, no work has examined the effects of RTM on the synthetic control method
nor on the statistical power of inference based on matched DID methods.

To build intuition for how matched DID estimators are subject to RTM bias,
consider the following example. We are interested in how a proposed treatment
affects the trajectory of an outcome variable. A single unit which will
receive treatment and a number of control units are sampled. Bias due to RTM
can be introduced when there is variability in outcome measures and the
population from which the treated unit is drawn differs from the control
population. Suppose the pre-treatment outcome measurements for the control and
treatment populations are normally distributed with mean $\mu_{0}$ and
$\mu_{1}$ respectively. Without loss of generality, assume $\mu_{1}>\mu_{0}$.
The nearest-neighbor match for the treated unit is expected to be a control
unit with an observed pre-treatment measurement greater than its expected
value. Thus, even if the post-treatment outcome distributions for the treated
and control populations are equivalent to those of the pre-treatment period
(i.e. no treatment effect), the matched unit is expected to decrease upon
repeat measurement. This creates the false appearance of a treatment effect.
Specifically, because the outcome trajectory of the matched control is
expected to be decreasing, the parallel trends assumption is violated and
estimates of the ATT are invalid.

###  2.3 Matching procedures

####  2.3.1 Synthetic Control Method

The method of synthetic controls is provided in detail elsewhere [2], and so
we provide only a brief overview. Suppose we collect data on a single treated
unit and $n_{0}$ controls for a total of $n_{0}+1$ units. Without loss of
generality, let $i=1$ denote the treated unit and $\mathcal{C}$ denote the set
of indices for the control units. We collect $\tau$ outcome measurements
$Y_{i}=(Y_{i1},\ldots Y_{i\tau})$ on each unit. Suppose that treatment is
withheld until time $\tau_{0}$, such that $j\in\\{1,\ldots,\tau_{0}\\}$ denote
the pre-treatment period and $j\in\\{\tau_{0}+1,\ldots,\tau\\}$ compose the
post-treatment period. The synthetic control method entails finding a set of
non-negative weights such that, at each pre-treatment timepoint, the weighted
sum of the control units’ outcomes closely resembles those of the treated
unit. That is, we aim to select $w_{k}$, for $k\in\mathcal{C}$ such that
$Y_{1j}\approx\sum_{k\in\mathcal{C}}w_{k}Y_{kj}$ for
$j\in\\{1,\ldots,\tau_{0}\\}$ and $\sum_{k\in\mathcal{C}}w_{k}=1$. If weights
are chosen so that these equalities approximately hold, then the weighted sum
of the post-treatment control vectors can serve as a potential untreated
outcome vector for the treated unit.

####  2.3.2 Nearest Neighbor matching

Nearest neighbor matching is a procedure which matches the treated unit with
the single control unit which is “closest” to it in the pre-treatment period.
To be concrete, let $S$ be a set of pre-treatment outcome measurements
observed in a sample. Suppose we have another unit with pre-treatment vector
$s$ and we want to find the nearest neighbor match for $s$ over the set $S$.
The nearest neighbor match given a selected distance metric is the element of
$S$ which minimizes the distance from $s$ [5]. Note that different distance
metrics may result in different matches. In this paper, we consider two
implementations of nearest neighbor matching. The first method is based upon
the distance between pre-treatment outcome vectors as determined by the
$L_{2}$-norm, while the second uses the $L_{1}$ distance between coefficients
in an OLS regression of pre-treatment outcome measurements on time (i.e. pre-
treatment trend).

##  3 Simulations

To examine the effect of RTM bias, we simulate a single treated unit alongside
$n_{0}=40$ controls. For control units, eight outcome measurements are drawn
from a multivariate normal distribution with mean $\mu_{0}$, marginal variance
$\sigma^{2}=1$, and first order autoregressive (AR(1)) covariance structure
with correlation $\rho^{|t_{i}-t_{j}|}$ between outcome measurements at
$t_{i}$ and $t_{j}$. The treatment unit is simulated similarly, with mean
$\mu_{1}$ rather than $\mu_{0}$. For each simulated dataset we match our
treated unit to controls using the synthetic control method, nearest neighbor
based on the $L_{2}$-norm, and nearest neighbor based on pre-treatment trend.
For comparison, we provide an estimate of the treatment effect using the
unmatched DID. If we define
$\bar{Y}_{0j}=n_{0}^{-1}\sum_{k\in\mathcal{C}}Y_{kj}$ as the mean of the
control units’ outcomes at time $j$, then the unmatched DID is calculated as

$\hat{\theta}=\frac{1}{4}\left[\sum_{j=5}^{8}\left\\{Y_{1j}-\bar{Y}_{0j}\right\\}-\sum_{j=1}^{4}\left\\{Y_{1j}-\bar{Y}_{0j}\right\\}\right]$

For the nearest neighbor and synthetic control methods, the estimator for
treatment effect simply replaces $\bar{Y}_{0j}$ with the value of the matched
or synthetic control at time $j$.

###  3.1 Type I errors and bias

Our first set of simulations looked to determine how type I error rates are
affected by outcome level matching. Permutation tests were used to test the
null hypothesis of no treatment effect. For each unit $i=1,\ldots,n_{0}+1$, we
relabel the data so that individual $i$ is the treated unit and all other
units are in the control group. We estimate the treatment effect under each of
the relabellings to obtain $\hat{\theta}_{i}$ for $i=1,\ldots,n_{0}+1$. The
p-value for this test is given as
$p=n^{-1}\sum\mathbb{I}(|\hat{\theta}_{1}|\geq|\hat{\theta}_{i}|)$. For
sufficiently low p-value, we reject the null hypothesis of no treatment
effect. This test is equivalent to the ”placebo test” described in Abadie et
al. [2].

First, we fix $\mu_{0}=0$ and $\rho=0.5$, while varying the value of
$\mu_{1}$. The magnitude of bias induced by regression to the mean is directly
related to the difference between the expected value of the controls’ outcomes
and those of the treated. Because of increased bias, type I error rates should
increase as the distance between $\mu_{0}$ and $\mu_{1}$ grows. This effect is
seen in the left-hand side of Table 1. Type I error rates were based on $2000$
simulations where we reject the null hypothesis for p-value lower than $0.05$.
On the right-hand side, we provide the results from varying the value of
$\rho$, which controls the correlation between repeat measurements. For these
simulations $\mu_{1}$ is set to $5$. As the correlation increases, we observe
that the type I error rate decreases. Errors which are highly correlated are
less prone to large shifts towards the mean, resulting in less bias in the
short term. In both scenarios, the synthetic controls method exaggerates the
effects of RTM bias when compared to the nearest neighbor methods. Again,
because the synthetic control uses information from all control units, there
is less variance in the estimator. Using synthetic controls results in more
confidence in the biased estimate. In both scenarios, matching based on pre-
treatment linear trend did not increase type I error rates. This is consistent
with findings from Daw et al. [9] showing that this matching did not introduce
perceptible bias into estimates of the ATT.

Type I Error Rate: Varying $\mu_{1}$ |  | Type I Error Rate: Varying $\rho$
---|---|---
$\mu_{1}$ | Unmatched | SC | $NN_{1}$ | $NN_{2}$ |  | $\rho$ | Unmatched | SC | $NN_{1}$ | $NN_{2}$
1.00 | 0.05 | 0.16 | 0.09 | 0.05 |  | 0.00 | 0.05 | 0.40 | 0.29 | 0.06
2.00 | 0.05 | 0.31 | 0.19 | 0.05 |  | 0.25 | 0.04 | 0.39 | 0.29 | 0.05
3.00 | 0.05 | 0.35 | 0.25 | 0.05 |  | 0.50 | 0.04 | 0.36 | 0.27 | 0.05
4.00 | 0.05 | 0.35 | 0.26 | 0.05 |  | 0.75 | 0.05 | 0.25 | 0.18 | 0.05
5.00 | 0.04 | 0.33 | 0.25 | 0.05 |  | 0.90 | 0.05 | 0.18 | 0.13 | 0.05
Table 1: Type I error rates for the unmatched DID, synthetic control (SC),
nearest neighbor using the $L_{2}$-norm $(NN_{1})$, and nearest neighbor using
linear trends $(NN_{2})$.

###  3.2 Loss of power

Matching on pre-treatment outcomes can cause us to reject the null hypothesis
more often than desired when there is no treatment effect. However, when there
is a treatment effect, RTM can also result in decreased power. Imagine a
scenario where $\mu_{1}$ is greater than $\mu_{0}$. As we have seen, matched
control units are likely to be those with randomly high pre-treatment outcome
levels. If there is no treatment effect, then the post-treatment difference
between the treated and matched control unit is expected to increase because
the control unit’s outcome measurements are expected to decrease. However, if
there is a treatment effect which causes the treated unit’s outcome level to
decrease, then the difference in post-treatment measurements will likely not
be as large as when there is no treatment effect. Thus, when the treatment
effect is in the same direction as RTM, matching on outcome levels can make us
less likely to reject the null hypothesis when there is a treatment effect and
more likely when there is not. Likewise, when the treatment effect is in the
opposite direction of RTM bias, the effect of treatment is expected to be
exaggerated.

To illustrate this phenomena, we perform $2000$ simulations with $\mu_{0}=0$,
$\mu_{1}=5$, and $\rho=0.5$. Here, we induce a treatment effect, $\theta$. For
each additional time point in the treatment period, the expected outcome for
the treated unit increased by $\theta$. Note that for negative $\theta$, the
treatment effect and RTM are working in the same direction and decreased power
is a concern. Figure 1 provides rejection rates for the unmatched and
synthetic control procedure when $\theta$ is between $0$ and $-1.5$. As in the
previous simulations, we see that when there is no treatment effect, the
synthetic control method exhibits inflated type I error rates while the
unmatched data has appropriate rejection rates. As the treatment effect
increases, the rejection rate of the unmatched estimator’s power surpasses
that of the synthetic control method. These results indicate that depending on
the direction of the treatment effect in relation to the direction of RTM, the
synthetic control method can result in either conservative or anti-
conservative bias.

Figure 1: Empirical probability of rejecting the hypothesis that there is no
treatment effect $(\theta=0)$ as a function of $\theta$ using the unmatched,
synthetic control method (SC), nearest neighbor matching on $L_{2}$-norm
(NN1), and nearest neighbor matching on linear trend (NN2).

###  3.3 Correction and Sensitivity Analysis

Suppose $Y_{1},\ldots,Y_{T}$ are jointly normal random variables. By
properties of the multivariate normal distribution, for any given $i$ and $j$,
we have
$\mathbb{E}\left[Y_{i}|Y_{j}\right]=\mu_{i}+\Sigma_{ij}\Sigma_{jj}^{-1}(Y_{j}-\mu_{j})$.
If we know the mean at each time point and the covariance structure, then we
can use this representation to account for RTM bias in matched DID estimates.
To illustrate, suppose we are performing a DID analysis using a single treated
observation and a sample of control units. Using a one-to-one matching
technique, the treated unit $Y_{1}$ is matched with a control unit, $Y_{m}$,
based upon pre-treatment outcome levels to obtain an estimate of the ATT, say
$\hat{\theta}_{obs}$. This estimate can be conceptualized as the sum of the
effect due to RTM bias and the effect due to treatment. Our correction
technique subtracts the estimated effect of RTM, $\hat{\theta}_{rtm}$, from
the observed effect to obtain a bias-adjusted estimate of the ATT,
$\hat{\theta}_{adj}=\hat{\theta}_{obs}-\hat{\theta}_{rtm}$.

To obtain $\hat{\theta}_{rtm}$, define
$\hat{Y}_{ij}=\mu_{ij}+\Sigma_{j\tau_{0}}\Sigma{\tau_{0}\tau_{0}}^{-1}\left(Y_{i\tau_{0}}-\mu_{1\tau_{0}}\right)$
for $j>\tau_{0}$ and $i\in\\{1,m\\}$. Here, $\tau_{0}$ is the final pre-
treatment observation time and $\mu_{ij}=\mathbb{E}[Y^{0}_{i}(j)]$ is the
expected potential outcome level under no treatment for unit $i$ at post-
treatment time $j$. Thus, $\hat{Y}_{ij}$ is the expected observation for unit
$i$ at post-treatment time $j$ conditional on the final pre-treatment
observation assuming no treatment effect. These expected values incorporate
the effect of RTM. Calculating the DID using these expected values in place of
observed post-treatment values for the treated and matched control units
provides $\hat{\theta}_{rtm}$, which can be used to find $\hat{\theta}_{adj}$.
Note that for one-to-one matching we only need to calculate $\hat{Y}_{ij}$ for
the treated and matched units. For the synthetic control correction, we will
need to estimate this value for all units.

To generalize this adjustment for use with synthetic controls, first obtain
the synthetic control weights $w_{k}$ for $k\in\mathcal{C}$. Using these
weights, we construct a synthetic outcome vector $Y_{S}$, where
$Y_{Sj}=\sum_{k\in\mathcal{C}}w_{k}Y_{kj}$, and find the observed estimate of
the ATT, $\hat{\theta}_{obs}$. We can obtain the expected DID under RTM by
using an augmented synthetic control. This new synthetic control is
constructed using the same weights as before and replacing post-treatment
control measurements with the $\hat{Y}_{ij}$’s defined earlier. Call this
augmented control $\hat{Y}_{S}$ and calculate the expected DID under no
treatment effect, $\hat{\theta}_{rtm}$, by subtracting the mean difference in
observed pre-treatment outcomes of the treated unit and the synthetic control
unit from the mean difference in expected post-treatment outcomes.

As a proof of concept, we perform $2000$ simulations with outcomes drawn from
a multivariate normal distribution with AR(1) error structure. Here,
$\mu_{0}=0$, $\mu_{1}=1$, $\sigma^{2}=1$, $\rho=0.5$, and there is no
treatment effect. For each simulation, we test the null hypothesis of no
treatment effect using the permutation test described earlier, replacing
$\hat{\theta}_{i}$ with $\hat{\theta}_{i,adj}$. This correction was derived
under the assumption of multivariate normality. To test if it is robust to
deviations from this assumption, we also look for type I error rates when
outcomes are drawn from a multivariate $t$-distribution. Simulation results
are given in Table 2. When errors are normally distributed, the adjusted
synthetic control estimate of the ATT attains nominal type I error rates. We
note that error rates are inflated for $t$-distributed outcomes, particularly
for high levels of $\rho$. However, observed error rates are lower than those
obtained in Table 1 using the unadjusted synthetic control approach with
normally distributed errors.

Degrees of Freedom | $\rho=0.25$ | $\rho=0.50$ | $\rho=0.75$
---|---|---|---
$\infty$ (Normal) | 0.05 | 0.05 | 0.05
50 | 0.05 | 0.05 | 0.06
10 | 0.05 | 0.06 | 0.08
3 | 0.05 | 0.08 | 0.12
Table 2: Type I error rates for the adjusted DID estimator when normality
assumption is not satisfied. Errors come from a t-distribution with degrees of
freedom described.

In practice, we do not have access to the true values of $\mu_{1j}$,
$\mu_{0j}$, $\rho$, or $\sigma$. So, estimating $\hat{\theta}_{adj}$ is not
possible without making additional assumptions on the values of these
parameters. To address this problem, we propose treating these as sensitivity
parameters. By positing a range of values for these parameters and calculating
$\hat{\theta}_{adj}$ under each set, we can quantify how much our estimate of
the ATT may be affected by RTM. For example, suppose we have obtained an
estimate of the ATT from a matched DID analysis and have determined that the
effect is significant using a permutation test. We wish to determine if the
observed significance can be explained away by RTM. To do so, we select values
of the sensitivity parameters under which there is no treatment effect,
estimate $\hat{\theta}_{adj}$ using these, and then perform the permutation
test. If the sensitivity parameters must be extreme or implausible in order to
nullify significance, then our results are unlikely to have been the result of
RTM bias.

##  4 Reanalysis of smoking cessation data

To further understanding of our proposed sensitivity analysis, we reanalyze
data from Abadie et al. [2] concerning the effect of California’s Proposition
99 on smoking cessation. Proposition 99 was an anti-smoking initiative
ratified by California state voters in 1988. The act added a 25 cents per pack
tax on the sale of cigarettes, and earmarked tax revenue for use in health
care programs and anti-tobacco advertisements.

The purpose of the original analysis was to determine whether Proposition 99
was successful in decreasing tobacco consumption. The authors concluded that
the initiative decreased cigarette consumption in California by about 20 packs
per capita annually. This difference was found to be statistically significant
using a permutation test. However, their analysis was based upon the synthetic
control method, which we have shown induces bias in estimates of the DID. To
determine the robustness of the original findings, we will perform a
sensitivity analysis.

Variable | Effect size | Standard Error
---|---|---
Year | -1.28 | 0.16
log(GDP) | -5.85 | 6.54
Beer consumption | -0.03 | 0.10
$\%$ Age 15 to 24 | -118.97 | 100.00
Retail price of cigarettes | 0.02 | 0.012
Cigarette sales per capita (1975) | 0.11 | 0.08
Cigarette sales per capita (1980) | 0.44 | 0.17
Cigarette sales per capita (1988) | 0.38 | 0.11

Figure 2: Left: Coefficients from fit GEE with AR(1) working correlation
matrix. Right: Tobacco consumption (per capita cigarette consumption) in a
subset of states between 1970-2000. California highlighted in black, treatment
initiation indicated by dashed red line.

This analysis was based upon cigarette consumption rates in 39 states,
California and 38 controls. Eleven states which implemented large-scale
tobacco control programs or that raised taxes on cigarette sales were excluded
from the analysis. The right-hand side of Figure 2 provides a plot of
cigarette consumption rates between 1970 and 2000 for the included states are
provided.

As in Abadie et al., we use logged per capita GDP, the average retail price of
cigarettes within each state, beer consumption per capita, and the percentage
of the population aged 15 to 24 to construct our synthetic control unit.
Following the example of the original study we also included cigarette sales
in the pre-treatment years 1975, 1980, and 1988. These variables were chosen
specifically because they were correlated with outcome levels. However, this
has a similar effect to matching on outcome levels, making RTM bias a concern.
Without adjusting for induced RTM bias, we ended up with an estimated ATT of
$21.25$ and a p-value of $2/39\approx 0.05$. That is, in the period from 1989
to 2000, after Proposition 99 had been enacted, the unadjusted estimate
determined that Californians consumed about 20 fewer packs of cigarettes per
capita than they would have if the law had not taken effect.

Our proposed method for adjusting the DID relies on knowledge of the mean
model under no treatment effect for each group. To obtain plausible mean
models, we regress per capita cigarette sales on the variables previously used
in the synthetic control method. Using the generalized estimating equation
(GEE) framework with AR(1) working correlation matrix, we obtain linear mean
models for each state’s cigarette consumption. In the left-hand side of Figure
2, estimated effect sizes and standard errors corresponding to the different
predictors are provided. We see that there is a strong downwards trajectory in
cigarette consumption over time in our dataset. To determine the error
structure (i.e. variance and correlation parameters) for our correction, we
calculate the sample residual variance, $\hat{s^{2}}$, and the sample
correlation, $\hat{\rho}$, between adjacent residuals within each state.
Respectively these values were estimated as $14.6$ and $0.93$.

Next, we use these values to calculate the expected DID due to RTM. If
$g_{i}(j)$ is the fitted value from the GEE model for state $i$ at time $j$,
then we define
$\hat{Y}_{ij}=\hat{\rho}^{j-1988}\left(Y_{i,1988}-g_{1}(1988)\right)+g_{i}(j)$
for $j>1988$ and $i\in\\{1,\ldots,39\\}$. Using the estimated $\hat{Y}_{i}$
vectors, we obtain an RTM adjusted estimate of the ATT $20.76$ and an updated
$p$-value of $4/39$ or $0.10$. While adjustment has a small effect on our
estimate of the DID, our results are no longer significant at the $0.05$
level.

To illustrate how our sensitivity analysis would work, suppose we are
interested in significance at the $0.1$ level. Consider the set of potential
mean models for the treated unit defined by $g^{*}_{1}(j)=g_{1}(j)+\Delta$.
For the purposes of this illustration, imagine that when $\Delta=0$ as in the
correction above, the test is still significant at the $0.05$ level. If small
changes in $\Delta$ result in flipped significance, then the results are not
robust to RTM. We re-estimate $\hat{\theta}_{adj}$ for a range of $\Delta$.
When $\Delta=-1$, our adjustment leads to an estimate of $20.41$ and a p-value
of $4/39$ which is greater than $0.1$. Note that $\Delta$ shifts the mean
model for per capita cigarette consumption. Comparing with the scale of Figure
2, we see that this is a relatively minor shift. Because the sensitivity
parameter, $\Delta$, did not need to be large to cause our results to be
insignificant at the $0.1$ level, we have evidence that the results of the
original study are not robust to RTM bias.

##  5 Discussion

In this paper we have illustrated the effects of RTM bias on the matched DID
estimator. This extends upon work done by Daw et al. [9] showing the bias
induced by nearest neighbor matching. Here, we have shown how this bias also
occurs when using the method of synthetic controls and have provided
simulations showing the effect of this bias on type I error rates and power.
Our results suggest that practitioners should be more cautious of RTM bias
when using synthetic controls than when using nearest neighbor matching. The
added ”confidence” in the model, gained from utilizing information from all of
the control units, increased the type I error rate by a factor of two over the
nearest neighbor approach.

We have further built on this work by developing an approach to determine how
sensitive matched DID estimates may be to RTM bias. Sensitivity analyses are
recognized as a critical component of causal analyses which allow us to
determine how robust our results may be to deviations from assumptions. Using
our approach, we showed that results concerning the effect of Proposition 99
on cigarette consumption in California may be overstated. After adjusting for
plausible bias due to RTM, the results of the analysis in Abadie et al. [2]
were no longer significant.

In the future, it may be worthwhile to look for ways to correct for RTM bias
when we can not assume normality of errors. For $t$-distributed errors, we
noticed that type I error rates were slightly greater than desired
$\alpha$-levels. While the adjustment still performed better than the
unadjusted synthetic control estimator, we believe the method could be
improved upon. As a whole, we believe that when researchers apply matched DID
estimators, they should also provide evidence that their results are robust to
RTM bias, either by using our adjusted DID estimator or by providing a
sensitivity analysis.

## References

  * [1] Alberto Abadie.  Semiparametric Difference-in-Differences Estimators.  The Review of Economic Studies, 72(1):1–19, 2005.
  * [2] Alberto Abadie, Alexis Diamond, and Jens Hainmueller.  Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program.  Journal of the American Statistical Association, 105(490):493–505, 2010.
  * [3] Robert P Althauser and Donald Rubin.  Measurement Error and Regression to the Mean in Matched Samples.  Social Forces, 50(2):206–214, 1971.
  * [4] Joshua D Angrist and Jörn-Steffen Pischke.  Mostly Harmless Econometrics: An Empiricist’s Companion.  Princeton University Press, 2008.
  * [5] Sunil Arya, David M Mount, Nathan Netanyahu, Ruth Silverman, and Angela Y Wu.  An Optimal Algorithm for Approximate Nearest Neighbor Searching in Fixed Dimensions.  In Proc. 5th ACM-SIAM Sympos. Discrete Algorithms, pages 573–582, 1994.
  * [6] Sanjay Basu, David H Rehkopf, Arjumand Siddiqi, M Maria Glymour, and Ichiro Kawachi.  Health Behaviors, Mental Health, and Health Care Utilization among Single Mothers after Welfare Reforms in the 1990s.  American Journal of Epidemiology, 183(6):531–538, 2016.
  * [7] Charles C Branas, Rose A Cheney, John M MacDonald, Vicky W Tam, Tara D Jackson, and Thomas R Ten Have.  A Difference-in-Differences Analysis of Health, Safety, and Greening Vacant Urban Space.  American Journal of Epidemiology, 174(11):1296–1306, 2011.
  * [8] David Card and Alan B Krueger.  Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania.  American Economic Review, 90(5):1397–1420, 2000.
  * [9] Jamie R Daw and Laura A Hatfield.  Matching and Regression to the Mean in Difference-in-Differences Analysis.  Health Services Research, 2018.
  * [10] Esther Duflo.  Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment.  American Economic Review, 91(4):795–813, 2001.
  * [11] Susan M Dynarski.  Does Aid Matter? Measuring the Effect of Student Aid on College Attendance and Completion.  American Economic Review, 93(1):279–288, 2003.
  * [12] Caroline M Fichtenberg and Stanton A Glantz.  Association of the California Tobacco Control Program with Declines in Cigarette Consumption and Mortality from Heart Disease.  New England Journal of Medicine, 343(24):1772–1777, 2000.
  * [13] John I Gallin and Frederick P Ognibene.  Principles and Practice of Clinical Research.  Academic Press, 2012.
  * [14] Stanton A Glantz.  Changes in Cigarette Consumption, Prices, and Tobacco Industry Revenues Associated with California’s Proposition 99.  Tobacco Control, 2(4):311, 1993.
  * [15] Rita Hamad, Akansha Batra, Deborah Karasek, Kaja Z LeWinn, Nicole R Bush, Robert L Davis, and Frances A Tylavsky.  The Impact of the Revised WIC Food Package on Maternal Nutrition during Pregnancy and Postpartum.  American Journal of Epidemiology, 2019.
  * [16] Michael Hendryx and Benjamin Holland.  Unintended consequences of the clean air act: Mortality rates in appalachian coal mining communities.  Environmental Science & Policy, 63:1–6, 2016.
  * [17] Rose MC Kagawa, Alvaro Castillo-Carniglia, Jon S Vernick, Daniel Webster, Cassandra Crifasi, Kara E Rudolph, Magdalena Cerdá, Aaron Shev, and Garen J Wintemute.  Repeal of Comprehensive Background Check Policies and Firearm Homicide and Suicide.  Epidemiology, 29(4):494–502, 2018.
  * [18] Noémi Kreif, Richard Grieve, Dominik Hangartner, Alex James Turner, Silviya Nikolova, and Matt Sutton.  Examination of the Synthetic Control method for Evaluating Health Policies with Multiple Treated Units.  Health Economics, 25(12):1514–1528, 2016.
  * [19] Timothy L Lash, Matthew P Fox, Richard F MacLehose, George Maldonado, Lawrence C McCandless, and Sander Greenland.  Good Practices for Quantitative Bias Analysis.  International Journal of Epidemiology, 43(6):1969–1985, 2014.
  * [20] James M Robins, Andrea Rotnitzky, and Daniel O Scharfstein.  Sensitivity Analysis for Selection Bias and Unmeasured Confounding in Missing Data and Causal Inference Models.  In Statistical Models in Epidemiology, the Environment, and Clinical Trials, pages 1–94. Springer, 2000.
  * [21] Donald B Rubin.  Causal Inference using Potential Outcomes: Design, Modeling, Decisions.  Journal of the American Statistical Association, 100(469):322–331, 2005.
  * [22] Andrew M Ryan, James F Burgess Jr, and Justin B Dimick.  Why we Should not be Indifferent to Specification Choices for Difference-in-Differences.  Health Services Research, 50(4):1211–1235, 2015.
  * [23] Michael Siegel.  The Effectiveness of State-Level Tobacco Control Interventions: a Review of Program Implementation and Behavioral Outcomes.  Annual Review of Public Health, 23(1):45–71, 2002.
  * [24] Diana Tordoff, Michele Andrasik, and Anjum Hajat.  Misclassification of Sex Assigned at Birth in the Behavioral Risk Factor Surveillance System and Transgender Reproductive Health: A Quantitative Bias Analysis.  Epidemiology, 30(5):669–678, 2019.

##  Appendix A Appendix

We wish to prove:

$D(t)=\mathbb{E}[Y^{1}(t)-Y^{0}(t)|A=1]-\beta(\mathbb{E}[X|A=1]-\mathbb{E}[Y|A=0])$

###### Proof.

Consider the following:

$\displaystyle D(t)$ $\displaystyle=\mathbb{E}[Y(t)|A=1]-\mathbb{E}[Y(t)|A=0]$
$\displaystyle=\mathbb{E}[Y^{1}(t)|A=1]-\mathbb{E}[Y^{0}(t)|A=0]$
$\displaystyle=\mathbb{E}[Y^{1}(t)-Y^{0}(t)|A=1]+\mathbb{E}[Y^{0}(t)|A=1]-\mathbb{E}[Y^{0}(t)|A=0]$

Here, the second line follows from the consistency assumption. Next note, for
$A=0$ or $1$, the expected value of the potential distribution can be
rewritten as:

$\displaystyle\mathbb{E}[Y^{0}(t)|A]$
$\displaystyle=\mathbb{E}\left\\{\mathbb{E}[Y^{0}(t)|X,A]|A\right\\}$
$\displaystyle=\mathbb{E}\left\\{\mathbb{E}[Y^{0}(t)|X]|A\right\\}$
$\displaystyle=\mathbb{E}[\beta X+\gamma t|A]$
$\displaystyle=\beta\mathbb{E}[X|A]+\gamma t$

The second line is true because we assume
$Y^{0}(t)\rotatebox[origin={c}]{90.0}{$\models$}A|X$. Plugging this into the
expression for $D(t)$ we can see,

$D(t)=\mathbb{E}[Y^{1}(t)-Y^{0}(t)|A=1]+\beta(\mathbb{E}[X|A=1]-\mathbb{E}[X|A=0])$

∎


# Exploring Dual-task Correlation for Pose Guided Person Image Generation

Pengze Zhang1, Lingxiao Yang1, Jianhuang Lai1,2,3 and Xiaohua Xie1,2,3
1School of Computer Science and Engineering, Sun Yat-Sen University, China
2Guangdong Province Key Laboratory of Information Security Technology, China
3Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of
Education, China
<EMAIL_ADDRESS>{yanglx9, stsljh<EMAIL_ADDRESS>Corresponding Author

###### Abstract

Pose Guided Person Image Generation (PGPIG) is the task of transforming a
person image from the source pose to a given target pose. Most of the existing
methods only focus on the ill-posed source-to-target task and fail to capture
reasonable texture mapping. To address this problem, we propose a novel Dual-
task Pose Transformer Network (DPTN), which introduces an auxiliary task
(i.e., source-to-source task) and exploits the dual-task correlation to
promote the performance of PGPIG. The DPTN is of a Siamese structure,
containing a source-to-source self-reconstruction branch, and a transformation
branch for source-to-target generation. By sharing partial weights between
them, the knowledge learned by the source-to-source task can effectively
assist the source-to-target learning. Furthermore, we bridge the two branches
with a proposed Pose Transformer Module (PTM) to adaptively explore the
correlation between features from dual tasks. Such correlation can establish
the fine-grained mapping of all the pixels between the sources and the
targets, and promote the source texture transmission to enhance the details of
the generated target images. Extensive experiments show that our DPTN
outperforms state-of-the-arts in terms of both PSNR and LPIPS. In addition,
our DPTN only contains 9.79 million parameters, which is significantly smaller
than other approaches. Our code is available at:
https://github.com/PangzeCheung/Dual-task-Pose-Transformer-Network.

##  1 Introduction

Pose Guided Person Image Generation (PGPIG) aims to generate person images
with arbitrary given poses. It has various applications such as e-commerce,
film special effects, person re-identification[40, 41, 34, 19, 35, 6, 5, 7],
etc. Due to the significant changes in texture and geometry during the pose
transfer, PGPIG is still a challenging task.

Figure 1: Visual comparison of our method with other approaches, including
vanilla CNN based [22], attention based [45], optical flow based [24], and
parsing map based [21] method. Compared with other methods, our model can
generate more realistic images.

Driven by the improvement of generative models, e.g., Generative Adversarial
Networks (GANs) [8] and Variational Autoencoders (VAEs) [17], PGPIG has made
great progress. However, early works [22, 4] are built on vanilla
Convolutional Neural Networks (CNNs), which lack the capability to perform
complex geometry transformations [13] (see Fig. 1 (c)). To tackle this
problem, attention mechanisms [45, 30] and optical flow [18, 24, 29] are
applied to improve spatial transformation abilities. Some methods [39, 21]
introduce additional labels such as human parsing maps to provide semantic
guidance for pose variations. However, the above mentioned methods solely
focus on training the generator $G$ on the Source-to-Target Task that
transforms the source image $\bm{x_{s}}$ from the source pose $\bm{p_{s}}$ to
the target pose $\bm{p_{t}}$:
$G(\bm{x_{s}},\bm{p_{s}},\bm{p_{t}})=\bm{\tilde{x}_{t}}$. This is an ill-posed
problem, making it arduous to train a robust generator. Moreover, the existing
methods cannot well capture the reasonable texture mapping between the source
and target images, especially when the person undergoes large pose changes.
Therefore, those methods often produce unrealistic images, as shown in Fig. 1
(d)-(f).

In this paper, we seek to utilize an auxiliary task [26] to improve the ill-
posed source-to-target transformation. Here, we instantiate the auxiliary task
as the Source-to-Source Task, which reconstructs the source image guided by
source pose: $G(\bm{x_{s}},\bm{p_{s}},\bm{p_{s}})=\bm{\tilde{x}_{s}}$. We
observe that simultaneously learning the dual tasks (i.e., source-to-target
task and source-to-source task) has the following two benefits: (1) Compared
with the source-to-target task, the pixel-aligned source-to-source task is
easier to learn because it does not require complex spatial transformations.
By sharing weights between the dual tasks, the source-to-source task can not
only exploit its knowledge to assist the source-to-target task, but also
stabilize the training of the whole network. (2) Since the intermediate
features in dual tasks are associated with their generated images
$\bm{\tilde{x}_{s}}$ and $\bm{\tilde{x}_{t}}$ respectively, we can further
explore the correlation between the dual tasks to establish the texture
transformation from the sources to the targets. In this way, the natural
source textures can be readily disseminated to enhance the details of the
generated target image.

Based on these ideas, we propose a novel Dual-task Pose Transformer Network
(DPTN) for PGPIG. The architecture of DPTN is shown in Fig. 2. Specifically,
our DPTN is of a Siamese structure, incorporating two branches: a self-
reconstruction branch for the auxiliary source-to-source task and a
transformation branch for the source-to-target task. These two branches share
partial weights, and are trained simultaneously with different loss functions.
By this means, the knowledge learned by the source-to-source task can directly
assist the optimization of the source-to-target task. To explore the
correlation between the dual tasks, we bridge the two branches with a novel
Pose Transformer Module (PTM). Our PTM consists of several Context Augment
Blocks (CABs) and Texture Transfer Blocks (TTBs). CABs first selectively
gather the information of the source-to-source task. Then TTBs gradually
capture the fine-grained correlation between the features from the dual tasks.
With the help of such correlation, TTBs can productively promote the texture
transmission from the real source image to the source-to-target task, enabling
the synthetic image to preserve more source appearance details. (see Fig. 1
(g)). In sum, the main contributions are:

  * •

We propose a novel Dual-task Pose Transformer Network (DPTN), which introduces
an auxiliary task (i.e., source-to-source task) by Siamese architecture and
exploits its knowledge to improve the PGPIG.

  * •

We design a Pose Transformer Module (PTM) to explore the dual-task
correlation. Such correlation can not only establish the fine-grained mapping
between the sources and the targets, but also effectively guide the source
texture transmission to further refine the feature in the source-to-target
task.

  * •

Results on the two benchmarks, i.e., DeepFashion [20] and Market-1501 [43],
have demonstrated that our method exhibits superior performance on PSNR and
LPIPS [42]. Moreover, our model only contains 9.79 million parameters, which
is relatively 91.6% smaller than the state-of-the-art method SPIG [21].

##  2 Related Works

Pose guided person image generation. Ma _et al._ [22] generated the fake image
in a coarse-to-fine manner. Esser _et al._ [4] combined the VAE and U-net [25]
to disentangle pose and appearance of the person. However, these methods are
based on vanilla CNNs, which cannot handle the complex deformation. To address
this problem, Zhu _et al._ [45] proposed a Pose Attention Transfer Network
(PATN) to optimize the appearance by pose relation. Furthermore, Tang _et al._
[30] added more crossing ways between the pose and appearance into PATN.
Nevertheless, these attention based methods do not explicitly learn the
spatial transformation between different poses, losing many source textures.

Figure 2: Overview of our model. It contains a self-reconstruction branch for
auxiliary source-to-source task, and a transformation branch for source-to-
target task. These two branches share partial weights and are communicated by
a pose transformer module.

To boost the texture transformation, Li _et al._ [18], Ren _et al._ [24] and
Tabejamaat _et al._ [29] proposed to introduce the warping operations to
PGPIG. They first estimated the dense optical flow, and then generated images
by warping the source image feature. Nevertheless, under the large pose change
and occlusion, these methods tended to produce inaccurate optical flow,
resulting in unsatisfied images.

Besides, both Zhang _et al._ [39] and Lv _et al._ [21] utilized additional
human parsing labels to improve the PGPIG. They first predicted the target
parsing maps, and then output person images with the help of semantic
information. However, the target parsing maps estimated by these methods are
often unreliable, which will mislead the generation of the synthetic images.
Moreover, pixel-wise annotations are hard to collect, which limits their
applications.

In summary, all the above methods only focus on the source-to-target task, and
cannot accurately capture the texture mapping between the source and the
target images. Contrary to them, we show that introducing the auxiliary
source-to-source task through a Siamese structure and simultaneously exploring
the dual-task correlation can further improve the performance of PGPIG.

Dual-task learning. Dual-task learning is a popular learning framework for
Natural Language Processing (NLP) [9, 37, 36], which utilizes the different
tasks to improve the learning progress. For example, [9] leveraged the closed-
loop of the English-to-French translation and French-to-English translation to
enhance each other, making it possible to train translation models without
paired data. Different from these methods, our dual tasks refer to the source-
to-source task and the source-to-target task. We have verified that the
source-to-source learning can promote the training of the source-to-target
task in PGPIG.

Transformers in vision tasks. Inspired by the success of transformers [33] in
NLP, many researchers had applied transformer architecture to computer vision
tasks such as image recognition [3, 31], object detection [2, 44], and image
generation [14, 12]. Specially, for image generation tasks, Jiang _et al._
[14] built a GAN with a pure transformer-based architecture without
convolutions. Hudson _et al._ [12] proposed a GANformer to exchange
information between image features and latent variables. However, these GANs
were designed for unconditional generation tasks, and were not well suitable
for conditional generation tasks with complex space deformation (i.e., PGPIG).
In this work, inspired by the core idea of the transformer, we design a novel
pose transformer module to explore the dual-task correlation.

##  3 Our Approach

Fig. 2 shows the overall framework of our DPTN. It mainly contains Siamese
branches for the dual tasks and a pose transformer module for exploring the
dual-task correlation. In the following sections, we will describe each
component of DPTN and loss functions in detail.

Table 1: The comparisons of the basic network on whether using the source-to-source learning. Both of the following two results are tested on the source-to-source task.  Learning scheme | PSNR $\uparrow$ | LPIPS $\downarrow$
---|---|---
Source-to-target learning | 19.1855 | 0.1962
\+ Source-to-source learning | 23.7606 | 0.1468
|  |

###  3.1 Siamese Structure for Dual Tasks

Figure 3: The structure of the Pose Transformer Module (PTM). It contains two
types of blocks: Context Augment Block (CAB) and Texture Transfer Block (TTB).
The CABs integrate the information of the feature $\bm{F_{s\rightarrow s}}$,
while the TTBs transfer the real source image textures from $\bm{F_{s}}$ to
optimize $\bm{F_{s\rightarrow t}}$ by capturing the correlation between
features from the dual tasks.

Although the existing PGPIG methods attempt to learn the source-to-target
transformation through various approaches, we argue that these methods ignore
some essential knowledge without the source-to-source learning, thus limiting
their potential improvement. To demonstrate this, we conduct an experiment on
a basic network (same structure as the self-reconstruction branch in Fig. 2,
including $En_{c}$, ResBlocks and $De$) to explore the impact of the source-
to-source learning, and show the results tested on the source-to-source task
in Tab. 1. Compared with source-to-target learning, the + source-to-source
learning in Tab. 1 only adds the self-reconstruction training, and does not
change the basic network structure. It can be seen that there is a significant
gap between these two learning schemes. Solely learning from the source-to-
target task cannot well reconstruct the source images, and lacks some
knowledge of PGPIG. Based on our analysis, in this paper, we add the source-
to-source task into PGPIG, and explore its knowledge to assist the source-to-
target transformation in the training process.

To achieve this goal, we construct our DPTN with a Siamese architecture,
incorporating two branches: a self-reconstruction branch for source-to-source
reconstruction, and a transformation branch for source-to-target generation.
As shown in Fig. 2, the two branches share three parts: an encoder $En_{c}$, a
series of ResBlocks, and a decoder $De$. In more detail, the encoder first
extracts the feature of two types of inputs, including the source-to-target
input (the concatenation of $\bm{x_{s}}$, $\bm{p_{s}}$ and $\bm{p_{t}}$) and
the source-to-source input (the concatenation of $\bm{x_{s}}$, $\bm{p_{s}}$
and $\bm{p_{s}}$). Then, ResBlocks are applied to gradually perform pose
transformation. Outputs of ResBlocks are the feature $\bm{F_{s\rightarrow s}}$
aligned with the source pose, and the transformed feature $\bm{F_{s\rightarrow
t}}$ aligned with the target pose. Finally, the $De$ in the self-
reconstruction branch accepts the $\bm{F_{s\rightarrow s}}$ to generate fake
source image $\bm{\widetilde{x}_{s}}$, and the $De$ in the transformation
branch accepts the refined feature $\bm{F^{*}_{s\rightarrow t}}$ (output of
our pose transformer module) to produce the target generated image
$\bm{\widetilde{x}_{t}}$.

In conclusion, the proposed Siamese architecture has the following advantages:
(1) Our encoder, ResBlocks and decoder are shared by the dual tasks, so that
the learned knowledge can be easily transferred between these tasks. (2)
Introducing self-reconstruction branch does not significantly add extra
parameters, as most of our model are reused in different tasks. (3) The
Siamese architecture enables the intermediate outputs of the dual tasks close
in feature distribution, facilitating the PTM in the next section to explore
the dual-task correlation.

###  3.2 Pose Transformer Module

In our Siamese structure, we have already obtained feature
$\bm{F_{s\rightarrow s}}$ aligned with the source pose $\bm{p_{s}}$, and
$\bm{F_{s\rightarrow t}}$ aligned with the target pose $\bm{p_{t}}$
respectively. However, since the vanilla CNN based transformation branch
(i.e., source-to-target) is hard to handle complex space deformation,
$\bm{F_{s\rightarrow t}}$ tends to lose many source appearance details, as
shown in Fig. 7. To tackle this problem, we propose a novel Pose Transformer
Module (PTM), which can further refine $\bm{F_{s\rightarrow t}}$ via capturing
the pixel-wise source-to-target correspondence between the features from dual
tasks. Our PTM is built upon the Multi-Head Attention (MHA) mechanism. To be
self-contained, we briefly introduce MHA as follows:

${Attention(\bm{Q},\bm{K},\bm{V})=softmax(\bm{Q}\bm{K}^{T}/{\sqrt{d_{k}}})\bm{V}},$
(1)
${\bm{head_{i}}=Attention(\bm{Q}\bm{W^{i}_{q}},\bm{K}\bm{W^{i}_{k}},\bm{V}\bm{W^{i}_{v}})},$
(2) ${MHA(\bm{Q},\bm{K},\bm{V})=concat(\bm{head_{1}},...,\bm{head_{h}})}.$ (3)

The $\bm{Q}$, $\bm{K}$, $\bm{V}$ are queries, keys and values.
${\bm{W^{i}_{q}}}$, ${\bm{W^{i}_{k}}}$, ${\bm{W^{i}_{v}}}$ are learnable
parameters. $h$ is the number of attention heads. $d_{k}$ is the dimension of
the keys. In particular, when $\bm{Q}=\bm{K}$, the MHA functions as Multi-Head
Self-Attention (MHSA); otherwise it acts as Multi-Head Cross-Attention (MHCA).

The proposed PTM is shown in Fig. 3. Unlike traditional vision transformer
[3], our PTM adopts a new architecture to explore the relation among triple
features (i.e., feature from source-to-source task, source-to-target task and
source image texture), making it more suitable for PGPIG. In general, PTM
contains two types of blocks: Context Augment Block (CAB) and Texture Transfer
Block (TTB), which can be formulated as: ${\bm{F^{N}_{s\rightarrow
s}}=CAB(...CAB(\bm{F_{s\rightarrow s}})...)}$, ${\bm{F^{*}_{s\rightarrow
t}}=TTB(...TTB(\bm{F_{s\rightarrow t}},\bm{F^{N}_{s\rightarrow
s}},\bm{F_{s}})...,\bm{F^{N}_{s\rightarrow s}},\bm{F_{s}})}$. The superscript
denotes the index of the feature. $N$ is the number of the blocks.
$\bm{F_{s\rightarrow s}}\\!=\\!\bm{F^{0}_{s\rightarrow s}}$,
$\bm{F_{s\rightarrow t}}\\!=\\!\bm{F^{0}_{s\rightarrow t}}$, and
$\bm{F^{*}_{s\rightarrow t}}\\!=\\!\bm{F^{N}_{s\rightarrow t}}$. $\bm{F_{s}}$
is the source image feature obtained by an additional encoder $En_{s}$. In the
proposed PTM, the CABs gradually integrate the information of the feature
$\bm{F_{s\rightarrow s}}$ from the self-reconstruction branch and produce
$\bm{F^{N}_{s\rightarrow s}}$. Then, each TTB combines three kinds of
features: the source image texture feature $\bm{F_{s}}$, the integrated
source-to-source feature $\bm{F_{s\rightarrow s}}$, and the previous TTB
output $\bm{F_{s\rightarrow t}}$. This combination is achieved by an MHCA
module to capture the correlation among all inputs. Next, we will present the
structure of the CAB and TTB respectively.

####  3.2.1 Context Augment Block

The structure of the $i$-th CAB is shown in the right-top of Fig. 3. It first
applies an MHSA unit with residual connection to adaptively enhance the
contextual representation of the input feature ${\bm{F^{i-1}_{s\rightarrow
s}}}$:

${\bm{\tilde{F}^{i}_{s\rightarrow s}}=IN(\bm{F^{i-1}_{s\rightarrow
s}}+MHSA(\bm{F^{i-1}_{s\rightarrow s}},\bm{F^{i-1}_{s\rightarrow
s}},\bm{F^{i-1}_{s\rightarrow s}}))},$ (4)

where IN is the instance normalization [32]. Then a Multi-Layer Perceptron
(MLP) module with multiple fully connected layers is used to increase the
capacity in CAB:

${\bm{F^{i}_{s\rightarrow s}}=IN(\bm{\tilde{F}^{i}_{s\rightarrow
s}}+MLP(\bm{\tilde{F}^{i}_{s\rightarrow s}}))}.$ (5)

After $N$ CABs, we obtain the final refined feature $\bm{F^{N}_{s\rightarrow
s}}$ and add this feature into each TTB for source-to-target task.

####  3.2.2 Texture Transfer Block

The structure of the $i$-th TTB is shown in the right-bottom of Fig. 3. First,
MHSA is applied to selectively focus on the key information of the
transformation branch feature $\bm{F^{i-1}_{s\rightarrow t}}$:

${\bm{\tilde{F}^{i}_{s\rightarrow t}}=IN(\bm{F^{i-1}_{s\rightarrow
t}}+MHSA(\bm{F^{i-1}_{s\rightarrow t}},\bm{F^{i-1}_{s\rightarrow
t}},\bm{F^{i-1}_{s\rightarrow t}}))}.$ (6)

Then an MHCA unit is employed to build correlation of
$\bm{\tilde{F}^{i}_{s\rightarrow t}}$, $\bm{F^{N}_{s\rightarrow s}}$, and
$\bm{{F}_{s}}$. Specifically, we employ $\bm{\tilde{F}^{i}_{s\rightarrow t}}$
as queries and $\bm{F^{N}_{s\rightarrow s}}$ as keys to calculate the pixel-
wise similarity between the sources and the targets. With the aid of such
similarity, $\bm{F_{s}}$ is used as values in MHCA to transmit the real source
textures to refine $\bm{\tilde{F}^{i}_{s\rightarrow t}}$. This procedure can
be written as:

${\bm{\hat{F}^{i}_{s\rightarrow t}}=IN(\bm{\tilde{F}^{i}_{s\rightarrow
t}}+MHCA(\bm{\tilde{F}^{i}_{s\rightarrow t}},\bm{F^{N}_{s\rightarrow
s}},\bm{F_{s}}))}.$ (7)

In this way, $\bm{\hat{F}^{i}_{s\rightarrow t}}$ carries more real source
textures, which will foster the transformation branch to generate more
delicate patterns. Finally, similar to CAB, the $i$-th TTB output
$\bm{F^{i}_{s\rightarrow t}}$ is obtained as follows:

${\bm{F^{i}_{s\rightarrow t}}=IN(\bm{\hat{F}^{i}_{s\rightarrow
t}}+MLP(\bm{\hat{F}^{i}_{s\rightarrow t}}))}.$ (8)

After $N$ time TTB blocks, the final output feature $\bm{F^{N}_{s\rightarrow
t}}$ will be fed into the decoder $De$ to generate target image
$\bm{\tilde{x}_{t}}$.

###  3.3 Loss Functions

Our network contains two branches for the source-to-source task and source-to-
target task. Thus, the overall loss function can be simply formulated as:

${\mathcal{L}=\mathcal{L}_{s\rightarrow s}+\mathcal{L}_{s\rightarrow t}},$ (9)

where $\mathcal{L}_{s\rightarrow s}$ and $\mathcal{L}_{s\rightarrow t}$ stand
for the loss of the dual tasks respectively. Both of them contain an ${l_{1}}$
loss ${\mathcal{L}_{l_{1}}}$, a perceptual loss ${\mathcal{L}_{perc}}$ and a
style loss ${\mathcal{L}_{style}}$. In addition, we apply an additional
adversarial loss ${\mathcal{L}_{adv}}$ in the source-to-target task to produce
more realistic textures. In sum, $\mathcal{L}_{s\rightarrow s}$ and
$\mathcal{L}_{s\rightarrow t}$ can be written as:

${\mathcal{L}_{s\rightarrow
s}=\mathcal{\lambda}_{l_{1}}\mathcal{L}^{s}_{l_{1}}+\mathcal{\lambda}_{perc}\mathcal{L}^{s}_{perc}+\mathcal{\lambda}_{style}\mathcal{L}^{s}_{style}},$
(10) ${\mathcal{L}_{s\rightarrow
t}=\lambda_{l_{1}}\mathcal{L}^{t}_{l_{1}}+\lambda_{perc}\mathcal{L}^{t}_{perc}+\lambda_{style}\mathcal{L}^{t}_{style}+\lambda_{adv}\mathcal{L}_{adv}},$
(11)

where $\lambda_{l_{1}}$, $\lambda_{perc}$, $\lambda_{style}$ and
$\lambda_{adv}$ are the loss weights for the dual tasks. Specifically, the
${l_{1}}$ loss penalizes the ${l_{1}}$ distance between the generated image
and the ground truth:

${\mathcal{L}^{d}_{l_{1}}={\|{\bm{x_{d}}-\bm{\tilde{x}_{d}}}\|}_{1}},$ (12)

where $d\in\\{s,t\\}$ represents the source or the target data. The perceptual
loss [15] calculates the feature distance:

${\mathcal{L}^{d}_{perc}=\sum\limits_{i}{{\|{\phi_{i}(\bm{x_{d}})-\phi_{i}(\bm{\tilde{x}_{d}}})\|}_{1}}},$
(13)

where $\phi_{i}$ denotes the $i$-th feature from VGG network [28]. The style
loss [15] compares the style similarity between images:

${\mathcal{L}^{d}_{style}=\sum\limits_{j}{{\|{\mathop{Gram}\nolimits_{j}^{\phi}(\bm{x_{d}})-\mathop{Gram}\nolimits_{j}^{\phi}(\bm{\tilde{x}_{d}}})\|}_{1}}},$
(14)

where $\mathop{Gram}^{\phi}_{j}$ is the Gram matrix of feature $\phi_{j}$.
Finally, the adversarial loss with a discriminator $D$ is employed to penalize
the distribution difference between the generated target image
${\bm{\tilde{x}_{t}}}$ and the ground truth ${\bm{{x}_{t}}}$:

${\mathcal{L}_{adv}=\mathbb{E}[\log(1-D(\bm{\tilde{x}_{t}}))]+\mathbb{E}[\log
D(\bm{x_{t}})]}.$ (15)

Table 2: Quantitative comparisons of image quality and model size with several state-of-the-art methods. * denotes the method using additional human parsing labels. The best and second best results are shown in bold and underline respectively. Model | DeepFashion | Market1501 |  Number of Parameters $\downarrow$
---|---|---|---
9 |  |  |  |  |  |  |  |
| SSIM $\uparrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$ |
PG2 [22] (NeurIPS’17) | 0.7730 | 17.5324 | 49.5674 | 0.2928 | 0.2704 | 14.1749 | 86.0288 | 0.3619 | 437.09 M
VU-net [4] (CVPR’18) | 0.7639 | 17.6582 | 15.5747 | 0.2415 | 0.2665 | 14.4220 | 44.2743 | 0.3285 | 139.36 M
DSC[27] (CVPR’18) | 0.7682 | 18.0990 | 21.2686 | 0.2440 | 0.3054 | 14.3081 | 27.0118 | 0.3029 | 82.08 M
PATN [45] (CVPR’19) | 0.7717 | 18.2543 | 20.7500 | 0.2536 | 0.2818 | 14.2622 | 22.6814 | 0.3194 | 41.36 M
DIAF [18] (CVPR’19) | 0.7738 | 16.9004 | 14.8825 | 0.2388 | 0.3052 | 14.2011 | 32.8787 | 0.3059 | 49.58 M
DIST [24] (CVPR’20) | 0.7677 | 18.5737 | 10.8429 | 0.2258 | 0.2808 | 14.3368 | 19.7403 | 0.2815 | 14.04 M
XingGAN [30] (ECCV’20) | 0.7706 | 17.9226 | 39.3194 | 0.2928 | 0.3044 | 14.4458 | 22.5198 | 0.3058 | 42.77 M
PISE∗ [39] (CVPR’21) | 0.7682 | 18.5208 | 11.5144 | 0.2080 | — | — | — | — | 64.01 M
SPIG∗ [21] (CVPR’21) | 0.7758 | 18.5867 | 12.7027 | 0.2102 | 0.3139 | 14.4894 | 23.0573 | 0.2777 | 117.13 M
Ours | 0.7782 | 19.1492 | 11.4664 | 0.1957 | 0.2854 | 14.5207 | 18.9946 | 0.2711 |  9.79 M
|  |  |  |  |  |  |  |  |

##  4 Experiments

###  4.1 Implementation Details

We evaluate our proposed model on two datasets: DeepFashion [20] and
Market1501 [43]. The DeepFashion dataset contains 52,712 high quality in-shop
clothes images (256 $\times$ 176) with clean backgrounds, while the Market1501
dataset contains 32,668 low-resolution images (128 $\times$ 64) with various
illumination and viewpoints. For a fair comparison, we split the datasets with
the same setting as [45]. It collects 101,966 training pairs and 8,570 testing
pairs for DeepFashion, and 263,632 training pairs and 12,000 testing pairs for
Market1501. In addition, the human pose keypoints are extracted from Human
Pose Estimator (HPE) [1].

In our experiment, Adam optimizer [16] is adopted to train the proposed DPTN
with the learning rate 1e-4. We choose $h=2$ and $N=2$ in the PTM on both
datasets. For the loss functions in Eq. 10 and Eq. 11, we set
$\mathcal{\lambda}_{l_{1}}=2.5$, $\mathcal{\lambda}_{perc}=0.25$,
$\mathcal{\lambda}_{style}=250$, $\mathcal{\lambda}_{adv}=2$.

###  4.2 Metrics

Following previous works [45, 21], we adopt Structural Similarity Index
Measure (SSIM) [38], Peak Signal-to-Noise Ratio (PSNR), Fréchet Inception
Distance (FID) [11] and Learned Perceptual Image Patch Similarity (LPIPS) [42]
as evaluation metrics. Moreover, we use rank-k and Mean Average Precision
(MAP) to further test the texture consistency between the source images and
the generated target images through the state-of-the-art re-identification
(re-id) platform FastReID [10]. More precisely, we train a re-id model on the
training set. Then we use the generated images as the query set and the real
images as the gallery set to calculate the metrics. High rank-k and MAP
indicate that the generated images do not lose much source appearance, and can
be easily recognized by the current re-id system.

###  4.3 Comparison with Previous Work

####  4.3.1 Quantitative Comparison

We compare our method with several state-of-the-art methods, including PG2
[22], VU-net [4], DSC[27], PATN [45], DIAF [18], DIST [24], XingGAN [30], PISE
[39] and SPIG [21]. Tab. 2 shows the quantitative results on image quality and
model size. As one can see, our method achieves seven best and one second-best
results among all compared methods, including PISE and SPIG using additional
parsing labels. This verifies the superiority of our DPTN in generating high
quality images. In addition, our DPTN only contains 9.79 M parameters, which
is 91.6% lower than that of SPIG (117.13 M). It clearly demonstrates the
efficiency of our method in modeling pose transformations.

Tab. 3 provides the comparison of the texture consistency on DeepFashion.
First, we train the re-id system on the DeepFashion training set. As shown in
the last row of Tab. 3, this re-id system achieves 99.08% rank-1 score. Then,
the same re-id system is applied to identify the person in the fake images
generated by different methods. From the result, we can find that our method
surpasses others in all four metrics. In particular, we promote the best
rank-1 performance of previous works by 3%. This indicates that the images
generated by our DPTN can effectively maintain the discriminative texture of
the source person.

Figure 4: Qualitative comparison with several state-of-the-art methods on DeepFashion (Left) and Market1501 (Right). Table 3: Quantitative comparisons of texture consistency. The best results are shown in bold. Methods | rank-1 $\uparrow$ | rank-5 $\uparrow$ | rank-10 $\uparrow$ | MAP $\uparrow$
---|---|---|---|---
PG2 | 60.12% | 75.44% | 81.95% | 59.20%
VU-net | 73.49% | 87.49% | 91.97% | 72.01%
DSC | 94.17% | 98.19% | 99.08% | 90.40%
PATN | 74.35% | 87.95% | 92.37% | 73.17%
DIAF | 94.87% | 98.02% | 99.13% | 91.45%
DIST | 90.84% | 96.64% | 98.11% | 87.56%
XingGAN | 59.63% | 72.48% | 81.19% | 58.36%
PISE | 90.09% | 96.35% | 98.02% | 87.22%
SPIG | 94.43% | 98.23% | 99.04% | 91.60%
Ours | 97.69% | 99.35% | 99.63% | 95.04%
Real Data | 99.08% | 99.80% | 99.88% | 98.40%
|  |  |  |

####  4.3.2 Qualitative Comparison

The qualitative comparison results are shown in Fig. 4. For the DeepFashion
dataset, the attention based methods PATN and XingGAN tend to generate blurred
images (see 1st and 2nd rows). DIAF and DIST attempt to promote the texture
transfer by using optical flow. However, in the case of large pose changes,
their predicted optical flow fails to represent such complex deformation,
resulting in unacceptable results (see 2nd and 3rd rows). PISE and SPIG
introduce the additional semantic parsing map to ease the difficulty of PGPIG.
Nevertheless, the target parsing maps estimated by these methods are often
inaccurate, which will mislead the generation of the synthetic images. For
example, in the 4th row on the left of Fig. 4, PISE and SPIG improperly
generate jackets in the synthetic images. Unlike the aforementioned methods,
our DPTN optimizes the source-to-target task with the help of the auxiliary
source-to-source task, making the generated image more vibrant. On the
Market1501 dataset, our DPTN can still generate finer and more vivid textures
than other methods. For instance, in the 4th row on the right of Fig. 4, only
our method retains the garment pattern of the source image.

###  4.4 Ablation Study

We conduct a series of experiments on DeepFashion to verify the contribution
of each component in our model. The various options for removing the
corresponding components from our full model are listed as follows.

The model without Dual-Task Learning (w/o DTL). This model is similar to the
existing methods that only focus on the source-to-target task. The entire
self-reconstruction branch, including $De$ and the loss function, is removed.

Figure 5: Qualitative comparison of the ablation study.

The model without Pose Transformer Module (w/o PTM). This model removes the
PTM. In this way, the source-to-target branch will lack the guidance of the
dual-task correlation, and will directly produce the target generated image
($\bm{\widetilde{x}_{t}}$) from $\bm{F_{s\rightarrow t}}$.

The model without Contextual Augment Blocks (w/o CABs). This model removes
CABs in the PTM. In this way, the feature from the source-to-source task
($\bm{F_{s\rightarrow s}}$) will be simply fed into TTBs to calculate the
dual-task correlation.

The model without encoder $\bm{En_{s}}$ (w/o $\bm{En_{s}}$). This model
removes the encoder $En_{s}$, and directly uses the feature
$\bm{F_{s\rightarrow s}}$ as the value in MHCA.

Full Model (Full). We use our proposed dual-task pose transformer network in
this model.

Table 4: Quantitative comparisons of ablation study on the DeepFashion dataset. The best results are shown in bold. Methods | SSIM $\uparrow$ | PSNR $\uparrow$ | FID $\downarrow$ | LPIPS $\downarrow$
---|---|---|---|---
w/o DTL | 0.7713 | 18.8134 | 14.7168 | 0.2143
w/o PTM | 0.7755 | 18.8503 | 15.5281 | 0.2195
w/o CABs | 0.7760 | 19.0489 | 12.0932 | 0.1989
w/o $En_{s}$ | 0.7778 | 19.1084 | 12.6858 | 0.1976
Full | 0.7782 | 19.1492 | 11.4664 | 0.1957
|  |  |  |
Figure 6: Learning curves of FID score by using source-to-target learning and
dual-task learning on DeepFashion dataset.

Fig. 5 and Tab. 4 show the qualitative and quantitative results of the
ablation study. As shown in Fig. 5, we can see that (1) Compared with the full
model, the model w/o DTL is unstable and tends to generate heavy artifacts.
This demonstrates the significance of the source-to-source task during the
training process. (2) Lack of texture mapping between the sources and the
targets, the model w/o PTM cannot well utilize the textures of the real source
image, resulting in blurred images. (3) For the model w/o CABs, the
information of the source-to-source task is not well integrated, which
misleads the source-to-target task to generate unrealistic patterns. (4) The
images generated by the model w/o $En_{s}$ lose many appearance details,
verifying the effect of $En_{s}$ in supplying fine source textures for PTM.
(5) Compared with others, our full model can not only generate satisfactory
global appearance but also produce realistic local textures. In addition, the
quantitative comparison in Tab. 4 further demonstrates the effectiveness of
our full model.

###  4.5 Effect of dual-task learning on training stability

To explore the influence of dual-task learning on training stability,
following [23], we visualize the learning curves of FID score under the
source-to-target learning and dual-task learning in Fig. 6. We can see that
the FID score of the DPTN with solely source-to-target learning plateaus
around 50 epochs, while the DPTN with dual-task learning continues to improve
even afterward. This verifies that the knowledge brought by the source-to-
source learning can effectively assist the learning of the source-to-target
task. In addition, compared with dual-task learning, the DPTN with solely
source-to-target learning tends to collapse after 130 epochs. This shows that
by sharing partial weights between the dual tasks, the training of the easier
source-to-source task can stabilize the training of the whole network to a
certain extent, so as to better optimize the source-to-target task.

###  4.6 Visualization of PTM

To explore how the PTM works in our framework, we also visualize the attention
weight in MHCA as well as the heatmaps of the $\bm{F_{s\rightarrow t}}$ and
$\bm{F^{*}_{s\rightarrow t}}$ in Fig. 7. As one can see, the attention weight
obtained in the PTM can accurately focus the area related to the query
position. This verifies that our PTM can effectively explore the pixel-wise
transformation between the sources and targets. In addition, compared with the
heatmap of the $\bm{F_{s\rightarrow t}}$, the $\bm{F^{*}_{s\rightarrow t}}$
produced by PTM contains more appearance cues. This manifests that our PTM can
transfer the natural source textures to refine $\bm{F_{s\rightarrow t}}$, and
facilitate the source-to-target task to generate more realistic details.

Figure 7: Visualization of the attention weight in MHCA and the heatmaps of
$\bm{F_{s\rightarrow t}}$ and $\bm{F^{*}_{s\rightarrow t}}$. The yellow star
on the fake image represents the query position.

##  5 Conclusions

In this paper, we propose a novel Dual-task Pose Transformer Network (DPTN)
for PGPIG. Unlike most of the existing methods only focusing on the source-to-
target task, our DPTN introduces an auxiliary task (i.e., source-to-source
task) by a Siamese architecture, and exploits its knowledge to assist the
source-to-target learning. Moreover, we carefully design a Pose Transformer
Model (PTM) to explore the correlation between the dual tasks. Such
correlation can be employed as a strong guidance for transferring source
textures to the target generated image. Both the quantitative and qualitative
results show that the proposed DPTN can improve upon prior PGPIG methods.

##### Acknowledgments.

This project is supported by the Key-Area Research and Development Program of
Guangdong Province (2019B010155003), and the National Natural Science
Foundation of China (62072482).

## References

  * [1] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 1302–1310, July 2017.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision (ECCV), pages 213–229, 2020.
  * [3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021\.
  * [4] Patrick Esser, Ekaterina Sutter, and Björn Ommer.  A variational u-net for conditional appearance and shape generation.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 8857–8866, June 2018.
  * [5] Zhanxiang Feng, Jianhuang Lai, and Xiaohua Xie.  Learning view-specific deep networks for person re-identification.  IEEE Transactions on Image Processing, 27(7):3472–3483, 2018.
  * [6] Zhanxiang Feng, Jianhuang Lai, and Xiaohua Xie.  Learning modality-specific representations for visible-infrared person re-identification.  IEEE Transactions on Image Processing, 29:579–590, 2020.
  * [7] Zhanxiang Feng, Jianhuang Lai, and Xiaohua Xie.  Resolution-aware knowledge distillation for efficient inference.  IEEE Transactions on Image Processing, 30:6985–6996, 2021.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems (NeurIPS), pages 2672–2680, 2014.
  * [9] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma.  Dual learning for machine translation.  In Advances in Neural Information Processing Systems (NeurIPS), pages 820–828, 2016.
  * [10] Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, and Tao Mei.  Fastreid: A pytorch toolbox for general instance re-identification.  arXiv preprint arXiv:2006.02631, 2020.
  * [11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems (NeurIPS), pages 6629–6640, 2017.
  * [12] Drew A Hudson and C. Lawrence Zitnick.  Generative adversarial transformers.  International Conference on Machine Learning (ICML), 2021.
  * [13] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu.  Spatial transformer networks.  In Advances in Neural Information Processing Systems (NeurIPS), pages 2017–2025, 2015.
  * [14] Yifan Jiang, Shiyu Chang, and Zhangyang Wang.  Transgan: Two transformers can make one strong gan.  arXiv preprint arXiv:2102.07074, 2021.
  * [15] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European Conference on Computer Vision (ECCV), pages 694–711, 2016.
  * [16] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations (ICLR), 2015\.
  * [17] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In International Conference on Learning Representations (ICLR), 2014\.
  * [18] Yining Li, Chen Huang, and Chen Change Loy.  Dense intrinsic appearance flow for human pose transfer.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 3688–3697, June 2019.
  * [19] Wenqi Liang, Guangcong Wang, Jianhuang Lai, and Xiaohua Xie.  Homogeneous-to-heterogeneous: Unsupervised learning for rgb-infrared person re-identification.  IEEE Transactions on Image Processing, 30:6392–6407, 2021.
  * [20] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 1096–1104, June 2016.
  * [21] Zhengyao Lv, Xiaoming Li, Xin Li, Fu Li, Tianwei Lin, Dongliang He, and Wangmeng Zuo.  Learning semantic person image generation by region-adaptive normalization.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 10806–10815, June 2021.
  * [22] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool.  Pose guided person image generation.  In Advances in Neural Information Processing Systems (NeurIPS), pages 405–415, 2017.
  * [23] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In International Conference on Learning Representations (ICLR), 2018\.
  * [24] Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H. Li, and Ge Li.  Deep image spatial transformation for person image generation.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 7687–7696, 2020.
  * [25] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234–241, 2015.
  * [26] Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  arXiv preprint arXiv:1706.05098, 2017.
  * [27] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuilière, and Nicu Sebe.  Deformable gans for pose-based human image generation.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 3408–3416, June 2018.
  * [28] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations (ICLR), 2015\.
  * [29] Mohsen Tabejamaat, Farhood Negin, and Francois Bremond.  Guided flow field estimation by generating independent patches.  British Machine Vision Conference (BMVC), 2021.
  * [30] Hao Tang, Song Bai, Li Zhang, Philip H. S. Torr, and Nicu Sebe.  Xinggan for person image generation.  In European Conference on Computer Vision (ECCV), pages 717–734, 2020.
  * [31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  arXiv preprint arXiv:2012.12877, 2020.
  * [32] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky.  Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 4105–4113, July 2017.
  * [33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NeurIPS), pages 6000–6010, 2017.
  * [34] Guangcong Wang, Jianhuang Lai, Peigen Huang, and Xiaohua Xie.  Spatial-temporal person re-identification.  Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):8933–8940, 2019.
  * [35] Guangcong Wang, Jianhuang Lai, and Xiaohua Xie.  P2snet: Can an image match a video for person re-identification in an end-to-end way?  IEEE Transactions on Circuits and Systems for Video Technology, 28(10):2777–2787, 2018.
  * [36] Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, Cheng Xiang Zhai, and Tie Yan Liu.  Multi-agent dual learning.  In International Conference on Learning Representations (ICLR), 2019\.
  * [37] Yijun Wang, Yingce Xia, Li Zhao, Jiang Bian, Tao Qin, Guiquan Liu, and Tie-Yan Liu.  Dual transfer learning for neural machine translation with marginal distribution regularization.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5553–5560, 2018.
  * [38] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [39] Jinsong Zhang, Kun Li, Yu-Kun Lai, and Jingyu Yang.  Pise: Person image synthesis and editing with decoupled gan.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 7982–7990, June 2021.
  * [40] Quan Zhang, Jianhuang Lai, Zhanxiang Feng, and Xiaohua Xie.  Seeing like a human: Asynchronous learning with dynamic progressive refinement for person re-identification.  IEEE Transactions on Image Processing, 31:352–365, 2022.
  * [41] Quan Zhang, Jianhuang Lai, and Xiaohua Xie.  Learning modal-invariant angular metric by cyclic projection network for vis-nir person re-identification.  IEEE Transactions on Image Processing, 30:8019–8033, 2021.
  * [42] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 586–595, June 2018.
  * [43] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1116–1124, 2015.
  * [44] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  In International Conference on Learning Representations (ICLR), 2021\.
  * [45] Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai.  Progressive pose attention transfer for person image generation.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition (CVPR), pages 2342–2351, 2019.