# Open-Set Face Identification
on Few-Shot Gallery by Fine-Tuning

Hojin Park, Jaewoo Park, and Andrew Beng Jin Teoh  School of Electrical and
Electronics Engineering
College of Engineering, Yonsei University
Seoul, Korea
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this paper, we focus on addressing the open-set face identification problem
on a few-shot gallery by fine-tuning. The problem assumes a realistic scenario
for face identification, where only a small number of face images is given for
enrollment and any unknown identity must be rejected during identification. We
observe that face recognition models pretrained on a large dataset and naively
fine-tuned models perform poorly for this task. Motivated by this issue, we
propose an effective fine-tuning scheme with classifier weight imprinting and
exclusive BatchNorm layer tuning. For further improvement of rejection
accuracy on unknown identities, we propose a novel matcher called Neighborhood
Aware Cosine (NAC) that computes similarity based on neighborhood information.
We validate the effectiveness of the proposed schemes thoroughly on large-
scale face benchmarks across different convolutional neural network
architectures. The source code for this project is available at:
https://github.com/1ho0jin1/OSFI-by-FineTuning

##  I Introduction

(a)

(b)

Figure 1: (a) Full fine-tuning all parameters severely degrades the OSFI
performance, while our method significantly improves the pre-trained model.
Detection & Identification Rate (DIR) [1] quantifies both correct
identification of the known probe identities and detection of the unknown. (b)
An outline of our proposed fine-tuning scheme: Given a model pretrained on a
large-scale face database, we initialize the gallery set classifier by weight
imprinting, and then fine-tune the model on a few-shot gallery set by training
only the BatchNorm layers. In the evaluation stage, a given probe is either
accepted as known or rejected as an unknown identity based on novel similarity
matcher dubbed Neighborhood Aware Cosine (NAC) matcher.

Recently face recognition (FR) has achieved astonishing success attributed to
three factors in large. Deep convolutional neural network (CNN) architectures
[2, 3] that have strong visual prior were developed and leveraged as FR
embedding models. Large-scale face datasets [4, 5] that cover massive
identities with diverse ethnicity and facial variations became available. On
top of these, various metric learning losses [6, 7, 8, 9] elevated the
performance of deep FR models to an unprecedented level.

The majority of FR embedding models have been evaluated on numerous benchmarks
with closed-set identification [7, 8, 9, 10, 11]. The closed-set
identification protocol assumes all probe identities present in the gallery.
However, in a realistic scenario, an unknown identity that is not enrolled may
be encountered. Another important but practical aspect to consider is the
scarcity of intra-class samples for the gallery identities to be registered;
namely, due to the expensive data acquisition cost and privacy issue, only a
very small number of samples might be available for each gallery identity to
register. In this respect, open-set face identification (OSFI) with the small-
sized gallery is closer to a real scenario as it needs to perform both known
probe identity identification and unknown probe identity rejection based on
the limited information from the small gallery set. Despite its versatile
practical significance, however, OSFI with a small gallery has been rarely
explored.

Devising a model specific to OSFI with a small gallery can be challenging in
the following aspects: Firstly, an OSFI model performs both identifications of
a known probe identity but also correct rejection of unknown probe identity.
Hence, conventional FR embedding models devised mainly for closed-set
identification can perform poorly at the rejection of the unknown. In fact, as
observed in Fig. 1 (a), FR embedding models pretrained on a large-scale public
face database are not effective for open-set identification, leaving room for
improvement. This suggests the need for fitting the pretrained model to be
more specific to the given gallery set.

Secondly, due to the few-shot nature of the small-sized gallery set, there is
a high risk of overfitting for fine-tuning the pretrained model. As shown in
Fig. 1 (a), full fine-tuning (i.e. updating all parameters) of the pretrained
model results in severe performance degradation. This drives us to devise an
overfitting-resilient parameter tuning scheme.

Moreover, an ordinary cosine similarity matcher used in the closed-set
identification might have a large tradeoff between the known probe identity
identification and unknown probe identity rejection. As will be observed in
Sec. III-D, the simple cosine matcher has a severe drawback for the task at
hand. This motivates us to devise a robust matcher for OSFI.

Based on these observations, we propose an efficient fine-tuning scheme and a
novel similarity-based matcher for OSFI constrained on a small gallery set.
Our fine-tuning scheme consists of weight initialization of the classifier
governed by weight imprinting (WI) [12] and training only BatchNorm (BN)
layers [13] for overfitting-resilient adaptation on the small gallery set.
Moreover, for both effective detection of the unknown and identification of
the known probe identities, a novel Neighborhood Aware Cosine (NAC) matcher
that respects the neighborhood information of the learned gallery features,
and hence better calibrates the rejection score is proposed. Our contributions
are summarized as follows:

  1. 1.

To effectively solve the OSFI problem constrained on a small gallery set, we
propose to fine-tune the pretrained face embedding model. Since full fine-
tuning deteriorates the embedding quality, we search for the optimal method.

  2. 2.

We demonstrate that the combination of weight imprinting and exclusive
BatchNorm layer fine-tuning excels other baselines.

  3. 3.

We recognize that the commonly used cosine similarity is a sub-optimal matcher
for rejection. We propose a novel matcher named NAC that significantly
improves the rejection accuracy.

##  II Related Works

###  II-A Open Set Face Identification (OSFI)

[14], one of the earliest works in OSFI, used their proposed Open-set TCM-kNN
on top of features extracted by PCA and Fisher Linear Discriminant. [15]
proposed their own OSFI protocol and showed that an extreme value machine [16]
trained on the gallery set performs better than using cosine similarity or
linear discriminant analysis for matchers. [17] trained a matcher composed of
locality sensitive hashing [18] and partial least squares[19]. [20] applied
OpenMax [21] and PROSER [22], two methods for open-set recognition of generic
images, on top of extracted face features.

All previous works propose to train an open-set classifier (matcher) of some
form, but all of them use a fixed encoder. To the best of our knowledge, we
are the first to propose an effective fine-tuning scheme as a solution to
OSFI.

###  II-B Cosine Similarity-based Loss Functions

[23] proposed to $l_{2}$-normalize the features such that the train loss is
only determined by the angle between the feature and the classifier weights.
[7] further extended this idea by applying a multiplicative margin to the
angle between a feature and its corresponding weight vector. This penalized
the intra-class features to be gathered while forcing inter-class centers
(prototypes) to be separated. A number of follow-up papers such as [8, 9, 10,
11] modify this angular margin term in different ways, but their motivations
and properties are generally similar. Therefore, in our experiments we only
use CosFace loss [8] as a representative method. For comprehensive
understanding of these loss functions, refer to [24].

##  III Approach

Our proposed approach is two-fold: fine-tuning on the gallery and open-set
identification evaluation. In the fine-tuning stage, the classifier is
initialized by weight imprinting to initiate learning from optimal
discriminative features, and the model is fine-tuned by updating only the
BatchNorm layers to avoid overfitting on the few-shot gallery data. In
evaluation, we utilize a novel matcher NAC that computes a neighborhood aware
similarity for better-calibrated rejection of the unknown. We demonstrate that
the combination of these three methods significantly outperforms all other
baselines.

###  III-A Problem Definition and Metrics

Formally, in an OSFI problem, we assume the availability of an encoder $\phi$
pretrained on a large-scale face database (FR embedding model), which is
disjoint from the evaluation set with respect to identity. The evaluation set
consists of a gallery $G=\\{(x^{G}_{i},y^{G}_{i})\\}^{Cm}_{i=1}$ and a probe
set $Q$. The probe set $Q$ is further divided into the known probe set
$K=\\{(x^{K}_{i},y^{K}_{i})\\}$ and the unknown probe set
$U=\\{(x^{U}_{i},y^{U}_{i})\\}$. $G$ and $K$ has no overlapping images $x$ but
shares same identities $y\in\\{1,...,C\\}$ whereas $U$ has disjoint
identities, i.e., $\mathcal{Y}^{U}\cap\\{1,...,C\\}=\O$. $m$ refers to the
number of images per identity in $G$, which we fix to 3 to satisfy the few-
shot constraint. We allow the encoder to be fine-tuned over the gallery set.

The evaluation of OSFI performance uses the detection and identification rate
at some false alarm rate (DIR@FAR). FAR=1 means we do not reject any probe.
Note that unlike the general case shown in [1], here we only consider rank-1
identification rate for DIR. Therefore, DIR@FAR=1 is the rank-1 closed-set
identification accuracy.

###  III-B Classifier Initialization by Weight Imprinting

Due to the few-shot nature of the gallery set where we fine-tune on, the
initialization of model parameters and, in particular, of classifier weights
is crucial to avoid overfitting. The most naive option is a random
initialization of the classifier weight matrix $W$. Another commonly used
strategy is linear probing [25], i.e., finding an optimized weight $W$ that
minimizes the classification loss over the frozen encoder embeddings
$\phi(x)$.

We experimentally find that, as seen in Fig. 2, both of these initialization
schemes do not induce discriminative structure for the encoder embedding
$\phi(x)$. In particular, during fine-tuning, each weight vector $w_{c}$ in
the classifier acts as a center (or prototype) for the $c$-th class (i.e.
identity). Fig. 2 shows that neither random initialization nor linear probing
of $w_{c}$ derives optimally discriminative weight vectors $w_{c}$, resulting
in low quality of class separation of gallery features.

Motivated from this issue, we propose to initialize by weight imprinting (WI),
which induces the optimal discriminative quality for the gallery features:

$w_{c}=\frac{\widehat{w_{c}}}{\lVert\widehat{w_{c}}\rVert_{2}},\quad\widehat{w_{c}}=\frac{1}{m}\sum_{y^{G}_{i}=c}\phi(x^{G}_{i})$
(1)

where $\lVert\cdot\rVert_{2}$ is the $l_{2}$ norm, and the embedding feature
$\phi(x)$ is unit-normalized such that $\lVert\phi(x)\rVert_{2}=1$.

As expected, Fig. 2 verifies that fine-tuning from the weight imprinted
initialization achieves a much higher discriminative quality. This shows the
superiority of weight imprinting compared to random initialization and linear
probing.

Note that weight imprinting has been frequently used in FR embedding models
[8, 9]. However, the critical difference is that those models utilize weight
imprinting only to prepare templates before evaluation. In our case, the WI
initialization is utilized particularly for fine-tuning.

###  III-C BatchNorm-only Fine-Tuning

Choosing the appropriate layer to tune is another important issue for fine-
tuning. Moreover, due to the extremely small number of samples for each
gallery identity, there is a risk of overfitting as suggested by the classical
theory on the vc dimension [26]. In fact, a recent study [25] suggests that
full fine-tuning hurts the pretrained filters including the useful
convolutional filters learned from a large-scale database.

To minimize the negative effect of this deterioration, we fine-tune only the
BatchNorm (BN) layers along with the classifier weight:

$\min_{W,\,\theta_{BN}}\mathcal{L}(W^{T}\phi_{\theta}(x),y),\quad\theta=[\theta_{BN},\theta_{rest}]$
(2)

where $\theta$ refers to all parameters in the encoder $\phi=\phi_{\theta}$
and $\theta_{BN}$ and $\theta_{rest}$ respectively refers to BatchNorm
parameters and the rest. During fine-tuning, $\theta_{rest}$ is fixed with no
gradient flow. The loss function $\mathcal{L}$ can be a softmax cross-entropy,
or widely used FR embedding model losses such as ArcFace [9] and CosFace [8].

Due to selective fine-tuning of only the BN layers (and classifier weight),
the convolutional filters learned from the large-scale pre-train database are
simply transferred. The BN-only training is thus computationally efficient as
it occupies only 0.1-0.01% of the total parameters in the CNN. Nevertheless,
its model complexity is sufficient to learn a general image task as guaranteed
by [27].

Figure 2: The Intra-class variance (left) and inter-class separation (right)
of classifiers that are initialized by different schemes. NormFace [23],
CosFace [8] and ArcFace [9] loss are used for linear probing initialization.
The weight imprinting initialization does not require training, thus stays
constant.

Figure 3: An unknown feature $u$ placed between gallery prototypes of class $i$ and $j$. $\epsilon$ is some small positive constant. TABLE I: Average angle (degrees) between IJB-C probe feature vectors and their top-k closest gallery prototypes. The third column refers to the average of top-2 to top-16. Encoder |  | top-1 | top-2 |  2$\sim$16
---|---|---|---|---
Res50 | K |  50.7∘ |  64.0∘ |  69.1∘
U |  63.8∘ |  66.0∘ |  69.7∘
VGG19 | K |  53.4∘ |  66.2∘ |  71.4∘
U |  65.9∘ |  68.2∘ |  72.1∘

###  III-D Neighborhood Aware Cosine Similarity

The cosine similarity function is the most predominant matcher for
contemporary face verification and identification. Denoting the probe feature
vector as $p$ and the gallery prototypes as $\\{g_{j}\\}_{j=1}^{C}$, where
$g_{j}:=\frac{1}{m}\sum_{y^{G}_{i}=j}\phi(x^{G}_{i})$ is the mean of all the
normalized gallery feature vectors of class $j$, identification is performed
by finding the maximum class index $c=\arg\max_{j=1,\dots,C}\cos(p,g_{j})$. On
the other hand, in the extension to OSFI, the decision of accepting as known
or rejecting as unknown can be formulated:

$\max_{j=1,\dots,C}\cos(p,g_{j})\underset{\text{Reject}}{\overset{\text{Accept}}{\gtrless}}\tau$
(3)

where $\cos(p,q)=\frac{p}{\lVert p\rVert_{2}}\cdot\frac{q}{\lVert
q\rVert_{2}}$ is the cosine similarity between two feature vectors, $\tau$ is
the rejection threshold.

Now, consider an example illustrated in Fig. III-C. The cosine matcher will
assign the probe $u$ to the identity $i$ with the acceptance score $0.866$,
which is fairly close to the maximum score $1$. This value alone might imply
that the probe is a known sample as it is close to the gallery identity $i$.
However, the probe feature vector is placed right in the middle of the
identities $i$ and $j$. The in-between placement of $u$ suggests that the
probe can be possibly unknown and thus should be assigned with a lesser value
of the acceptance score.

Motivated by this intuition, we propose the Neighborhood Aware Cosine (NAC)
matcher that respects all top-$k$ surrounding gallery features:

$\text{NAC}(p,g_{i})=\frac{\exp(cos(p,g_{i}))\cdot\text{1}[i\in
N_{k}]}{\sum_{j\in N_{k}}\exp(\cos(p,g_{j}))}$ (4)

Here, $N_{k}$ is the index set of $k$ gallery prototypes that are nearest to
the probe feature $p$, and 1 is the indicator function. The main goal of the
NAC matcher is to improve the unknown rejection. Table III-C shows that known
probe features are much closer to their closest prototype than the second-
closest prototype, unlike unknown probes. By exploiting this phenomenon, the
NAC matcher is able to assign a much smaller score to unknown probe, as shown
in Fig. 4.

Figure 4: The distributions of scores for known (K) and unknown (U) probes of
IJB-C dataset using cosine similarity (left) and NAC with $k=16$ (right). The
scores are min-max normalized and $\tau$ is set such that FAR=0.01 for both
cases. DIR=48.05% (left) vs DIR=54.53% (right). ResNet-50 was used as the
encoder.

##  IV Experiments

###  IV-A Datasets

We use VGGFace2 [4] dataset for pretraining the encoders, and CASIA-WebFace
[28] and IJB-C [29] for evaluation. Using MTCNN [30], we align and crop every
images to 112x112 with equal parameters for all datasets. For VGGFace2, we
remove all identities overlapping with the evaluation datasets. The evaluation
datasets are equally split into two groups such that the number of known and
unknown identities are equal. Then we randomly choose $m{=}3$ images of the
known identities to create the gallery (G), and the rest are known probes (K).
All images of unknown identities are unknown probes (U). Table II summarizes
the statistics of the datasets we use. Note that we chose every known identity
to have more than 10 images such that there can be at least 7 probe samples.
Also note that IJB-C dataset consists of still images and video frames (video
frames typically have poorer image quality). We sample the gallery from still
images and probes from video frames, which makes this dataset much
challenging. We note that the protocol devised here can be regarded as an
extension of that in [15].

###  IV-B Baselines

####  IV-B1 Classifier Initialization

Along with Weight Imprinting (denoted WI), we report the results of using
random initialization and linear probing initialization as described in Sec.
III-B.

####  IV-B2 Encoder Layer Fine-Tuning

Along with BatchNorm-only fine-tuning (denoted as BN), we explore tuning other
layers of the encoder. The simplest one is tuning every layer (i.e. all
parameters of a model), which we denote as full. The second is freezing the
early layers and training only the deeper ones, which we denote as partial. We
also consider the parallel residual adapter [31], which adds additional 1x1
convolutional filters to the original convolutional layers. During fine-
tuning, only these additional filters are trained to capture the subtle
difference in the new dataset. Note that the authors in [31] apply this
technique to ResNet [3], hence the name residual parallel adapter. But this
idea can be generally applied to CNNs without residual connection, hence we
also apply this to a VGG-style network. We denote this as PA, referring to
Parallel Adapter.

TABLE II: Dataset Statistics. The number inside the parentheses refers to the average number of images per identity. For evaluation datasets, Known identities consist of the gallery (G) and known probe (K), where the gallery has 3 images per identity. Pretrain | # IDs (images / ID)
---|---
VGGFace2 | 7,689 (354.0)
Evaluation | Known (G + K) | Unknown (U)
CASIA-WebFace | 5,287 (3+20.0) | 5,288 (16.5)
IJB-C | 1,765 (3+15.3) | 1,765 (13.9)
TABLE III: The total number of parameters and number of fine-tuned parameters for each encoder fine-tuning scheme. ‘+’ refers to the number of added parameters for the Parallel Adapter. | # Params (million)
---|---
| VGG19 | Res50
Pretrained | 32.88 | 43.58
Full fine-tuning | 32.88 | 43.58
Partial fine-tuning | 4.72 | 4.72
Parallel Adapter | +2.22 | +3.39
BN-only fine-tuning | 0.01 | 0.03

####  IV-B3 Matcher

During OSFI evaluation, the vanilla cosine similarity matcher is adopted as
the baseline matcher. When the NAC matcher is used, we denote by NAC. For
comparison, we also use the extreme value machine (EVM) proposed by [15]. We
train the EVM on the gallery set with the best parameters found by the
authors.

In summary, classifier initialization methods we consider are {Random, Linear
probing, WI}, fine-tuning layer configurations are {Full, Partial, PA, BN},
and matchers are {cos, EVM, NAC}. We test the OSFI performances among
different combinations of these three components. Our proposed OSFI scheme is
to use WI+BN+NAC jointly.

Figure 5: The OSFI performance of cosine similarity and NAC with different
values of $k$ on IJB-C dataset, using VGGNet-19 (left) and ResNet-50 (mid) as
the encoder. The square markers refer to cosine similarity and star marks the
optimal k for different layer fine-tuning methods. To summarize the OSFI
performance into a single number, we used the area under the curve (AUC, %) of
DIR@FAR curve. (Right) DIR@FAR curve of Pretrained and BN configuration using
cosine similarity and NAC (k=16) as the matcher. Numbers in the legend show
the AUC values. When $k=1$, NAC is replaced by $\cos$.

###  IV-C Training Details

We choose VGG19 [2] and ResNet-50 [3] for the encoders with the feature
dimension 512. We pretrain these encoders on the VGGFace2 dataset with CosFace
with scale=32, margin=0.4 as loss function until convergence.

Then we fine-tune the encoder with different classifier initialization schemes
and encoder layer configurations. When using the linear probing
initialization, we train the classifier until the training accuracy reaches
95%.

We follow the encoder layer finetuning in Sec. IV-B. For the partial fine-
tuning, we only train the last 2 convolutional layers (Conv-BN-ReLU-Conv-BN-
ReLU). Table III shows the number of total and updated parameters for each
fine-tuning scheme.

We fix the number of epochs to 20 and batch size to 128 for every method. We
again use CosFace loss for consistency. For the optimizer we use Adam [32]
with cosine annealing. The initial learning rate is set to 1e-4 for full and
PA, and 1e-3 for partial and BN, which we find as the optimal learning rate
for each method. For data augmentation, we use random horizontal flipping and
random cropping with the random scale from 0.7 to 1.0. The cropped images are
resized to the original size.

TABLE IV: DIR@FAR of different methods on CASIA-WebFace dataset and IJB-C dataset, using VGGNet-19 and ResNet-50 as the encoder. DIR@FAR=1 (100%) is the closed-set accuracy. The highest value in each column is marked in bold. For the first three rows the encoder is not fine-tuned and only the matchers are changed. The last row (WI+BN+NAC) is our proposed method. Encoder | Method | CASIA-WebFace | IJB-C
---|---|---|---
Classifier initialization |  Fine-tuning layers | Matcher | DIR @ FAR (%) | DIR @ FAR (%)
0.1 | 1.0 | 10.0 | 100.0 | 0.1 | 1.0 | 10.0 | 100.0
VGG19 | None | None | cos | 25.23 | 52.97 | 70.07 | 80.89 | 28.35 | 45.55 | 61.71 | 73.80
None | None | EVM | 37.57 | 57.75 | 71.03 | 80.78 | 35.03 | 53.64 | 63.34 | 73.70
None | None | NAC | 25.15 | 55.68 | 71.41 | 80.89 | 36.73 | 51.92 | 64.27 | 73.80
Random | Full | cos | 23.95 | 43.19 | 59.03 | 70.94 | 17.18 | 32.62 | 46.90 | 60.23
Linear probing | Full | cos | 28.82 | 55.64 | 70.44 | 79.84 | 30.80 | 45.91 | 59.63 | 70.09
WI | Full | cos | 27.63 | 57.58 | 72.02 | 80.94 | 35.49 | 50.52 | 63.56 | 73.53
WI | Partial | cos | 28.91 | 57.31 | 72.29 | 81.16 | 34.81 | 51.98 | 64.53 | 73.89
WI | PA | cos | 26.29 | 57.90 | 72.82 | 81.82 | 31.74 | 50.21 | 64.26 | 74.50
WI | BN | cos | 25.39 | 56.65 | 72.54 | 82.14 | 32.19 | 48.74 | 63.87 | 74.43
WI | BN | NAC | 25.94 | 58.01 | 72.92 | 82.14 | 38.09 | 53.08 | 65.30 | 74.43
Res50 | None | None | cos | 23.85 | 58.06 | 74.15 | 83.69 | 32.11 | 48.05 | 65.31 | 76.96
None | None | EVM | 39.44 | 61.61 | 75.02 | 83.57 | 38.12 | 38.12 | 66.81 | 76.96
None | None | NAC | 21.24 | 60.23 | 75.31 | 83.69 | 36.67 | 54.53 | 68.14 | 76.96
Random | Full | cos | 25.31 | 45.43 | 60.80 | 72.44 | 14.88 | 32.05 | 49.39 | 61.88
Linear probing | Full | cos | 28.35 | 60.11 | 74.63 | 82.73 | 30.35 | 46.42 | 61.90 | 72.34
WI | Full | cos | 26.73 | 63.92 | 77.49 | 84.65 | 39.05 | 56.00 | 67.83 | 76.94
WI | Partial | cos | 25.98 | 64.66 | 78.07 | 85.02 | 44.31 | 57.11 | 69.13 | 77.49
WI | PA | cos | 24.89 | 63.85 | 77.58 | 85.01 | 36.69 | 54.86 | 68.30 | 77.63
WI | BN | cos | 25.70 | 65.83 | 79.66 | 86.73 | 40.29 | 55.71 | 69.29 | 78.74
WI | BN | NAC | 23.65 | 67.72 | 80.34 | 86.73 | 40.25 | 58.25 | 70.40 | 78.74

###  IV-D Optimal k for NAC

Since the gallery set is too small, we cannot afford a separate validation set
to individually optimize $k$ for each dataset. Instead, we attempt to find a
global value that has optimal performance regardless of the fine-tuning
method, if one exists.

We first fine-tune the encoders with different layer configurations, which
gives us five different encoders including one without any fine-tuning;
pretrained, full, partial, PA, and BN. Then we search the best parameter k for
the NAC matcher by grid search strategy, where the grid is
[2,4,8,16,32,128,256,512,1024,$C$], and $C$ is the total number of identities.
Note that $k=1$ refers to using cosine similarity instead of NAC, which we
added for comparison. Since a single-value objective is preferred, we use the
area under the curve (AUC) of the DIR@FAR curve instead of DIR value at
different FAR values. We repeat this process with different datasets and
encoder architectures.

The results are shown in Fig. 5. We did not include the results of CASIA-
WebFace as it shows a similar trend. Excluding $k=1$ which is not NAC, the
results show a smooth unimodal curve with a peak at $k=16$ or $32$. This shows
that the NAC matcher indeed has a globally optimal $k$ value that is robust
against different datasets, encoders, and fine-tune methods. Thus we choose
$k=16$ ($k=32$ also gives similar results) as the global parameter throughout
this paper.

Note that when $k=C$, NAC becomes equivalent to softmax function with cosine
similarity logits. However, this is notably inferior compared to $k=16$, which
implies that considering only the $k$-nearest is superior to considering every
gallery prototype.

###  IV-E Comparison of Fine-Tuning Methods

We compare the OSFI performances of the pretrained model (non-fine-tuned) with
six different combinations of classifier initialization schemes and layer
finetuning configurations: random+full, linear probing+full, WI+full,
WI+partial, WI+PA, WI+BN. The matcher is fixed to cosine similarity. These
correspond to row 4-9 in Table IV.

First, to compare different classifier initialization schemes, we fix the
fine-tuning scheme to full. When using random initialization, rejection
accuracy (DIR@FAR=0.001,0.01,0.1) and closed-set accuracy (DIR@FAR=1) severely
drops. For linear probing, rejection accuracy improves while closed-set
accuracy drops. Only WI clearly improves the encoder performance, supporting
the superiority of weight imprinting.

Now we fix the classifier initialization to WI and compare different layer
finetuning configurations. full clearly has the worst performance. While PA is
better than partial in closed-set accuracy, partial clearly outperforms PA in
rejection accuracy. BN outperforms all others in closed-set accuracy with a
large margin but sometimes falls behind partial in rejection accuracy.

With the aid of the NAC matcher, our method WI+BN+NAC outperforms all other
methods in every aspect. Compared to original, this gains 4.60%, 8.11%, 4.57%,
1.68% higher DIR in average with respect to FAR of 0.001, 0.01, 0.1, 1.0,
respectively.

TABLE V: Inter-class separation, intra-class variance, DBI, and AUC gain by using NAC (refer to Fig. 5) for each layer finetuning configuration. These values are averaged across datasets and encoder architectures. $\uparrow$ means that larger quantity is better and vice versa. | Inter ($\uparrow$) | Intra ($\downarrow$) | DBI ($\downarrow$) |  $\Delta$AUC ($\uparrow$)
---|---|---|---|---
Pretrained Model | 106.3∘ | 34.5∘ | 1.52 | 0.740
Full finetuning | 106.7∘ | 24.2∘ | 0.87 | 0.025
Partial finetuning | 106.4∘ | 24.5∘ | 0.90 | 0.058
Parallel Adapter | 107.0∘ | 31.8∘ | 1.32 | 0.135
BN-only finetuning | 107.3∘ | 33.6∘ | 1.46 | 0.335

###  IV-F Analysis on Discriminative Quality of Different Fine-tuning Methods

How do different layer finetuning configurations affect the final OSFI
performance? To analyze this, we adopt three different metrics; inter-class
separation, intra-class variance, and Davies-Bouldin Index (DBI) [33]. The
definitions of the first two metrics are identical to that of Fig. 2. DBI is a
metric for evaluating the clustering quality, where DBI $\approx 0$ means
perfect clustering. We compute these metrics on the gallery features after
fine-tuning, and the results are shown in Table V.

Here we can easily separate these configurations into two groups: full and
partial vs PA and BN. The first group has similar inter-class separation with
Pretrained and significantly smaller intra-class variance, which leads to
small DBI. This is in stark contrast with the second group.

With this observation, we can conjecture the different optimization strategies
of each group. The first group was able to easily reduce the training loss by
collapsing the gallery features into a single direction (shown by the small
angle between intra-class features). This was possible because both full and
partial directly updated the parameters of the convolutional filters. On the
other hand, all convolutional filters were frozen for both PA and BN. This
constraint may have prevented these methods from taking the shortcut, i.e.
simply collapsing the gallery features, and instead led to separating the
embeddings of different identities. This explains why PA and BN have higher
closed-set accuracy.

This can also explain the AUC gain ($\Delta$AUC) when using NAC instead of
cosine similarity. Features become redundant when they collapse, and so does
the prototype. Therefore the information from neighboring prototypes becomes
less helpful in rejecting unknown samples, leading to the marginal gain from
using NAC. This is why full and partial do not benefit from using NAC matcher.

Figure 6: The performance of our method against the baseline w.r.t. different
gallery size. AUC of DIR@FAR curve is used as the performance measure.

###  IV-G Performance with respect to Different Gallery Size

Fig. 6 shows the OSFI performance of our method against the baseline
(pretrained encoder with cos matcher) with respect to different gallery size.
We can see that our method consistently improves upon the baseline, except for
the extreme case where only one image is provided for each identity.

##  V Conclusion and Future Works

In this work we showed that combining weight-imprinted classifier and
BatchNorm-only tuning of the encoder effectively improves the encoder’s OSFI
performance without suffering from overfitting. We further facilitated the
performance by our novel NAC matcher instead of the commonly used cosine
similarity. Future works will explore extending this idea to the open-set few-
shot recognition of generic images.
Acknowledgements:
This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIP) (NO. NRF-2022R1A2C1010710)

## References

  * [1] A. K. Jain and S. Z. Li, _Handbook of face recognition_. Springer, 2011, vol. 1.
  * [2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [4] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in _2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018)_. IEEE, 2018, pp. 67–74.
  * [5] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao, “Ms-celeb-1m: A dataset and benchmark for large-scale face recognition,” in _European conference on computer vision_. Springer, 2016, pp. 87–102.
  * [6] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 815–823.
  * [7] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 212–220.
  * [8] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “Cosface: Large margin cosine loss for deep face recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5265–5274.
  * [9] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4690–4699.
  * [10] X. Wang, S. Zhang, S. Wang, T. Fu, H. Shi, and T. Mei, “Mis-classified vector guided softmax loss for face recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 12 241–12 248.
  * [11] Q. Meng, S. Zhao, Z. Huang, and F. Zhou, “Magface: A universal representation for face recognition and quality assessment,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 225–14 234.
  * [12] H. Qi, M. Brown, and D. G. Lowe, “Low-shot learning with imprinted weights,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 5822–5830.
  * [13] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [14] F. Li and H. Wechsler, “Open set face recognition using transduction,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 27, no. 11, pp. 1686–1697, 2005.
  * [15] M. Gunther, S. Cruz, E. M. Rudd, and T. E. Boult, “Toward open-set face recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 71–80.
  * [16] E. M. Rudd, L. P. Jain, W. J. Scheirer, and T. E. Boult, “The extreme value machine,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 3, pp. 762–768, 2017.
  * [17] R. Vareto, S. Silva, F. Costa, and W. R. Schwartz, “Towards open-set face recognition using hashing functions,” in _2017 IEEE international joint conference on biometrics (IJCB)_. IEEE, 2017, pp. 634–641.
  * [18] B. Kulis and K. Grauman, “Kernelized locality-sensitive hashing,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 34, no. 6, pp. 1092–1104, 2011.
  * [19] G. Mateos-Aparicio, “Partial least squares (pls) methods: Origins, evolution, and application to social sciences,” _Communications in Statistics-Theory and Methods_ , vol. 40, no. 13, pp. 2305–2317, 2011.
  * [20] H. Dao, D.-H. Nguyen, and M.-T. Tran, “Face recognition in the wild for secure authentication with open set approach,” in _International Conference on Future Data and Security Engineering_. Springer, 2021, pp. 338–355.
  * [21] A. Bendale and T. E. Boult, “Towards open set deep networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1563–1572.
  * [22] D.-W. Zhou, H.-J. Ye, and D.-C. Zhan, “Learning placeholders for open-set recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4401–4410.
  * [23] F. Wang, X. Xiang, J. Cheng, and A. L. Yuille, “Normface: L2 hypersphere embedding for face verification,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 1041–1049.
  * [24] I. Masi, Y. Wu, T. Hassner, and P. Natarajan, “Deep face recognition: A survey,” in _2018 31st SIBGRAPI conference on graphics, patterns and images (SIBGRAPI)_. IEEE, 2018, pp. 471–478.
  * [25] Anonymous, “Fine-tuning distorts pretrained features and underperforms out-of-distribution,” in _Submitted to The Tenth International Conference on Learning Representations_ , 2022, under review. [Online]. Available: https://openreview.net/forum?id=UYneFzXSJWh
  * [26] V. N. Vapnik and A. Y. Chervonenkis, “On the uniform convergence of relative frequencies of events to their probabilities,” in _Measures of complexity_. Springer, 2015, pp. 11–30.
  * [27] J. Frankle, D. J. Schwab, and A. S. Morcos, “Training batchnorm and only batchnorm: On the expressive power of random features in cnns,” _arXiv preprint arXiv:2003.00152_ , 2020.
  * [28] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” _arXiv preprint arXiv:1411.7923_ , 2014.
  * [29] B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney _et al._ , “Iarpa janus benchmark-c: Face dataset and protocol,” in _2018 International Conference on Biometrics (ICB)_. IEEE, 2018, pp. 158–165.
  * [30] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.
  * [31] S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Efficient parametrization of multi-domain deep neural networks,” 2018.
  * [32] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [33] D. L. Davies and D. W. Bouldin, “A cluster separation measure,” _IEEE transactions on pattern analysis and machine intelligence_ , no. 2, pp. 224–227, 1979.


# VITON-DiT: Learning In-the-Wild Video Try-On from Human Dance Videos via
Diffusion Transformers

Jun Zheng
Sun Yat-Sen University
<EMAIL_ADDRESS>
&Fuwei Zhao
ByteDance China
<EMAIL_ADDRESS>
&Youjiang Xu
ByteDance China
<EMAIL_ADDRESS>
&Xin Dong
ByteDance China
<EMAIL_ADDRESS>
&Xiaodan Liang
Sun Yat-Sen University
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Video try-on stands as a promising area for its tremendous real-world
potential. Prior works are limited to transferring product clothing images
onto person videos with simple poses and backgrounds, while underperforming on
casually captured videos. Recently, Sora revealed the scalability of Diffusion
Transformer (DiT) in generating lifelike videos featuring real-world
scenarios. Inspired by this, we explore and propose the first DiT-based video
try-on framework for practical in-the-wild applications, named VITON-DiT.
Specifically, VITON-DiT consists of a garment extractor, a Spatial-Temporal
denoising DiT, and an identity preservation ControlNet. To faithfully recover
the clothing details, the extracted garment features are fused with the self-
attention outputs of the denoising DiT and the ControlNet. We also introduce
novel random selection strategies during training and an Interpolated Auto-
Regressive (IAR) technique at inference to facilitate long video generation.
Unlike existing attempts that require the laborious and restrictive
construction of a paired training dataset, severely limiting their
scalability, VITON-DiT alleviates this by relying solely on unpaired human
dance videos and a carefully designed multi-stage training strategy.
Furthermore, we curate a challenging benchmark dataset to evaluate the
performance of casual video try-on. Extensive experiments demonstrate the
superiority of VITON-DiT in generating spatio-temporal consistent try-on
results for in-the-wild videos with complicated human poses. Project page:
https://zhengjun-ai.github.io/viton-dit-page/.

##  1 Introduction

Video virtual try-on systems [6, 51, 21, 43] aim to dress a target person in
video with desired clothing while maintaining their motion and identity. It
offers tremendous potential for practical uses such as e-commerce and
entertainment. While video representation is more compelling, it is also more
challenging. Therefore, the majority of existing work has focused on image-
based try-on [14, 5, 26, 10, 16, 40, 50, 41]. The earlier approaches typically
build on Generative Adversarial Networks (GANs) [40, 16, 5, 50, 41],
containing a warping module and a try-on generator. The warping module deforms
clothing to align with the human body, and then the warped garment is fused
with the person image through the try-on generator. However, with the recent
advent of UNet-based Latent Diffusion Models (LDMs) [32, 47, 28, 44],
researchers’ attention has gradually shifted to these emerging generative
models for more groundbreaking results. A diffusion-based try-on network does
not explicitly separate the warping and blending operations, but instead
unifies them into a single process of cross-attention implicitly. By
leveraging text-to-image pre-trained weights, these diffusion approaches
demonstrate superior fidelity compared to the GAN-based counterparts.

Figure 1: Video try-on results of the proposed VITON-DiT. Our model is capable
of generalization across diverse types of clothing, even those non-product
garments that may have flaws. It can also deal with complex body movements
such as dancing against real world backgrounds.

Despite promising image generation quality, UNet-based LDMs appear to fall
short when handling video scenes. Whereas the newfound Transformer-based LDMs
(or Diffusion Transformer, DiT) exhibit remarkable capability and scalability
in generating high-fidelity real-world images/videos, such as Stable Diffusion
3 [8] and Sora [29]. Inspired by Sora, we propose VITON-DiT, the first DiT-
based video try-on network tailored to tackle in-the-wild scenarios. Fig. 1
shows samples generated by our model111Please refer to the supplemented video
for more results.. Specifically, VITON-DIT contains three main components: a
spatio-temporal denoising DiT for video latent generation, a garment extractor
to maintain clothing details faithfully, and an ID ControlNet to preserve the
person’s pose and identity. These three modules are connected by an innovative
_attention fusion_ mechanism. This mechanism combines extracted garment
features with person denoising features via an addictive attention process,
enabling seamless integration of clothing characteristics into the video
generation process. Furthermore, we devise a random selection strategy during
training and an Interpolated Auto-Regressive (IAR) sampling technique at
inference to facilitate long video generation up to tens of seconds.

Existing video try-on methods rely heavily on paired datasets [6], i.e., a
product clothing image coupled with a video of a person wearing that clothing.
Constructing such datasets is labor-intensive, limiting scalability and
generalization ability. In contrast, our VITON-DiT only requires an unpaired
collection of human dance videos, facilitated by a multi-stage training
procedure. We also show that with the expansion of data volume, the model’s
performance scales up proportionately, highlighting its robust data
scalability. Moreover, we propose a new video try-on benchmark capturing real-
world complexities to comprehensively evaluate model performance.

Extensive experiments showcase VITON-DiT’s superiority in generating realistic
video try-on results, even for unconstrained scenes involving complex body
motions and backgrounds. Our model takes the first step towards video virtual
try-on for authentic real-world uses. Our main contributions include four
aspects:

  * •

We present the first DiT-based video try-on network, VITON-DiT, featuring
consistent spatio-temporal generation on casually captured human videos.

  * •

We propose an attention fusion algorithm to bridge the three modules of VITON-
DiT, enabling precise recovery of clothing details in videos.

  * •

We devise a novel random selection strategy and an interpolated auto-
regressive technique to foster long video generation.

  * •

Our model is scalable to unpaired human video data, and we provide a new
challenging video try-on benchmark to facilitate related research fields.

##  2 Related Work

Video Virtual Try-on. Existing work on video virtual try-on can be classified
as GAN-based [6, 51, 24, 21] and diffusion-based methods [43]. The former
relies on garment warping by optical flow [7] and utilizes a GAN generator to
fuse the warped clothing with the reference person. FW-GAN [6] predicts
optical flow to warp preceding frames during the video try-on process, thereby
ensuring the generation of temporally coherent video sequences. ClothFormer
[21] presents a dual-stream transformer architecture to efficiently integrate
garment and person features, facilitating more accurate and realistic video
try-on results. Despite reasonable performance, GAN-based methods struggle to
garment-person misalignment in case of inaccurate warping flow estimation. And
the overall generation quality is inferior to diffusion-based models that are
with large-scale pre-trained weights. The recent Tunnel Try-on [43] proposes
to use a UNet-based diffusion model for video try-on. It can handle camera
movements and faithfully preserve the clothing textures. However, their
provided demo videos are only for product images and span just a few seconds.
Instead, our VITON-DiT can be applied to diverse types of clothing even with
missing parts, and can generate tens of seconds of try-on sequences in in-the-
wild scenarios with high-quality spatiotemporal consistency.

Diffusion Models for Video Generation The success of text-to-image (T2I)
diffusion models leads to emerging studies on text-to-video synthesis. Such
studies [49, 38, 13, 12] typically insert additional temporal dimensions or
layers into the pre-trained T2I models. For example, Video LDM [1] proposes an
innovative two-stage training process, initially focusing on static images,
followed by a temporal layers specifically trained on video datasets.
Similarly, AnimateDiff [13] introduces a versatile plug-and-paly motion module
for existing T2I models, mitigating extensive model-specific adjustments.
However, these approaches still face challenges in long video generation due
to the restricted capacity and scalability of the UNet architecture, whereas
DiT-based T2V models [29, 17, 25] are capable of directly generating videos
with tens of seconds. The recent phenomenal Sora [29] adopts DiT as backbone
and can generate realistic 1-minute videos from texts. However, Sora is close-
sourced currently and we resort to one of its alternatives named Open-Sora
[17]. Our VITON-DiT is built upon OpenSora and served as the first DiT-based
model for video virtual try-on.

##  3 Method

We present VITON-DiT, a video virtual try-on framework built upon diffusion
transformers (DiT) [30]. Our model comprises three components as shown in Fig.
2(a): (1) the main Denoising DiT built by a series of _Spatio-Temporal (ST-)
DiT_ blocks, performing latent diffusion procedure and generate video try-on
results, (2) the Garment Extractor that processes and delivers garment
features to retain clothing details, (3) the trainable ID ControlNet  that
guides the Denoising DiT to preserve the target person’s pose and identity.
These three modules are connected by an operation called _attention fusion_ ,
as depicted in Fig. 2(b). It fuses the extracted garment features with the
person denoising features via summing up the attentive outputs. We find this
addictive operation works well for recovering garment details in the generated
videos. Before introducing our architecture, we briefly review some basic
concepts of Latent Diffusion Models (LDMs) and DiT in Sec. 3.1. The three
parts of VITON-DiT are elaborated from Sec. 3.2 to Sec. 3.4. At Sec. 3.5, we
further elucidate a novel _random agnostic condition swap_ during training and
an _interpolated auto-regressive (IAR) technique_ at inference. These
strategies lead to improved long video generation up to 20 seconds.

Figure 2: Overview of the proposed VITON-DiT. (a) The architecture contains
three components with the following tasks. (1) Denoising DiT: generating
latent representation of video contents via a chain of Spatio-Temporal (ST-)
DiT blocks. (2) ID ControlNet: producing feature residual for the Denoising
DiT to preserve the reference person’s identity, pose, and background. (3)
Garment Extractor: obtaining and delivering garment features into the
Denoising DiT and the ControlNet via attention fusion, thus recovering
detailed clothing textures in the generated try-on video. (b) Illustrated
Attention Fusion: integrating person denoising features and extracted garment
features using addictive attention. This operation is utilized in both the
Denoising DiT and the ID ControlNet.

###  3.1 Preliminary

Latent Diffusion Models (LDMs). Generating high-resolution images/videos
directly in the original pixel space can be computationally expensive and
challenging due to the high dimensionality. Instead, LDMs [32] operate in a
latent space where the data is represented in a more compact form. This
approach leverages the power of variational autoencoders (VAEs) [23] to encode
the high-dimensional data into a latent space and then apply the diffusion
process in this latent space. An image LDM typically contains three key
components: (a) an Encoder $\mathcal{E}$ mapping the high-resolution image $x$
to a latent representation $z=\mathcal{E}\left(x\right)$ , (b) a Diffusion
Process involving a forward process that gradually adds noise to $z$ over $T$
time steps:

$q\left(z_{t}\mid
z_{t-1}\right)=\mathcal{N}\left(z_{t};\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I\right),$
(1)

where $\beta_{t}$ is a variance schedule that controls the amount of noise
added at each step; and a reverse process parameterized by a neural network
(typically a U-Net [33]) $p_{\theta}$ that learns to denoise:

$p_{\theta}\left(z_{t-1}\mid
z_{t}\right)=\mathcal{N}\left(z_{t-1};\mu_{\theta}\left(z_{t},t\right),\sigma_{\theta}^{2}(t)I\right),$
(2)

(c) a Decoder $\mathcal{D}$ maps the denoised latent representation back to
the original image space: $\hat{x}=\mathcal{D}_{\left(z_{0}\right)}$. The
training objective is typically a reconstruction loss in the latent space that
minimizes the noise $\epsilon$ and the network’s prediction:

$L_{LDM}=\mathbb{E}_{z,\epsilon,t}\left[\left\|\epsilon-
p_{\theta}\left(z_{t},t\right)\right\|_{2}^{2}\right].$ (3)

Once Trained, we can sample $z_{t}$ from $p_{(z)}$ and decode it to image
space with a single pass through $\mathcal{D}$.

Diffusion Transformers (DiT). The Diffusion Transformer [30] is an innovative
architecture that leverages the strengths of diffusion models and transformers
[35]. By integrating these two powerful paradigms, it aims to extend the
quality, flexibility, and scalability of the traditional UNet-based LDMs [32].
The overall formulation remains the same as the LDMs except using a
transformer (instead of a UNet) to learn the denoising function $p_{\theta}$
within a diffusion-based framework. Inspired by OpenSora [17], we adopt a
modified Spatio-Temporal DiT (ST-DiT) as the backbone of our VITON-DiT. We
find the spatio-temporal disentangled transformers are particularly well-
suited for handling human video data with limited computational resources.

###  3.2 Spatio-Temporal Diffusion Transformer (ST-DiT)

As shown in Fig. 2(a), each ST-DiT block consists of three attention layers
that respectively perform Spatial Self-Attention (SSA), Temporal Self-
Attention (TSA), and Prompt Cross-Attention (PCA), following by a point-wise
feed-forward layer that bridges two adjacent ST-DiT blocks. The normalized
input of each layer is further skip-connected with the layer’s output.

Specifically, the input video $x\in\mathbb{R}^{f\times H\times W\times 3}$ is
first projected into the latent space via a fixed VAE encoder $\mathcal{E}$
[9], producing the video latent $z_{0}\in\mathbb{R}^{f\times h\times w\times
4}=\mathcal{E}\left(x\right)$ , where $h=H/8$, $w=W/8$, and $f$ refers to the
number of frames. Given patch size $p\times p$, the spatial represented
$z_{0}$ is then “patchified” into a sequence of length $s=hw/p^{2}$ with
hidden dimension $d$, forming the input token $I\in\mathbb{R}^{f\times s\times
d}$. Following patchify, the input video tokens $I$ are processed sequentially
by SSA, TSA and TSA. In SSA, a vanilla Attention
$(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot V$ is
applied on the spatial dimension of normalized $I$ with

$Q=W_{Q}\cdot I_{norm},K=W_{K}\cdot I_{norm},V=W_{V}\cdot I_{norm}.$ (4)

Here, $W_{Q},W_{K},W_{V}\in\mathbb{R}^{d\times d_{\epsilon}}$ are learnable
projection matrices [19, 35]. While in TSA, the self-attention is performed on
the temporal dimension $f$ to perceive and integrate dynamic information of
the input dance frames. Lastly, we send embedding of a default prompt “a
dancing person” through T5 text encoder [31] into the PCA layer, where a cross
attention is conducted between the prompt embedding and the intermediate
feature from the TSA.

The denoising ST-DiT plays a vital role in refining the generated video
sequences. The three attention layers in each block are tasked with texture
generation/preservation, keeping temporal consistency and enhancing overall
visual quality, respectively. Note during training, we load pre-trained
weights from OpenSora [17] and freeze the PCA & feed-forward layer to retain
its original generation ability. The parameters of SSA and TSA are trainable
as they need to process additional garment features or be adapted for in-the-
wild human videos. The training objective for the main denoising DiT
$\mathcal{P}$ is the standard latent diffusion loss:

$L_{\mathcal{P}}=\mathbb{E}_{z,\epsilon,t}\left[\left\|\epsilon-
p_{\theta}\left(z_{t},t\right)\right\|_{2}^{2}\right].$ (5)

###  3.3 Garment Extractor

The key to video try-on is recovering the texture details of the desired
garments in the generation results. To this end, we design a garment extractor
DiT $\mathcal{G}$ in parallel with the main denoising DiT, as illustrated by
Fig. 2(a). It drops the temporal attention as the input contains only a single
clothing image $c$ (i.e., without temporal information).

Similar to the denoising DiT, the clothing image is encoded by $\mathcal{E}$
and passes through $N$ garment extractor blocks. In each pass, we store the
intermediate features before being sent to the SSA layer, then input them into
the main DiT and the ID ControlNet. In particular, an Attention Fusion
depicted by Fig. 2(b) comes into play and associates the garment encoder with
the other two modules via additive attention.

Specifically, the extracted garment feature $r_{c}$ is firstly repeated along
the temporal dimension, matching the shape of the person denoising feature
$r_{p}$ from the same location in the denoising DiT and the ControlNet.
Secondly, $r_{p}$ and $r_{c}$ are run through the SSA and the Spatial Cross-
Attention (SCA), respectively. Lastly, the two output features are added and
sent to the next blocks (TSA, PCA, etc) as detailed in Sec. 3.2. The intuition
of this attention fusion is to incorporate garment features via SCA followed
by a summed injection. We formulate the attention fusion process as:

$\mathcal{F}\left(r_{p},r_{c}\right)=SSA(r_{p},r_{p})+SCA(r_{p},r_{c})$ (6)

A line of work [18, 37, 42, 3] has proved the effectiveness of this operation
in keeping the texture details. We differ from them in our newly inserted SCA
layer. By using an additional cross-attention layer instead of reusing the
same SSA layer, we improve the model’s capacity and capability to perceive the
garment features, thereby reducing the burden of SCA and eliminating its input
ambiguity.

###  3.4 Identity Preservation ControlNet (ID ControlNet)

At its core, video virtual try-on can be viewed as an inpainting problem. It
requires four-tuple $\left\\{x_{a},d_{p},m_{c},c\right\\}$ to put the target
clothing $c$ onto the reference person video $x$, including the cloth-agnostic
image $x_{a}$, DensePose image $d_{p}$, and the inpainting mask $m_{c}$, as
visualized in Fig. 2(a). As the pre-trained weight from OpenSora is not tuned
for inpainting, we introduce a DiT ControlNet to preserve the person’s pose,
identity and background, named ID ControlNet $\mathcal{C}$.

Specifically, the ID ControlNet is a trainable replica of the front half of
the denoising DiT. Similar to the U-Net ContorlNet [47], two zero-initialized
linear layers are plugged into the two ends of $\mathcal{C}$, as shown in Fig.
2(a). The difference here is the denoising DiT does not have skip-connections
between its internal blocks. We therefore add the outputs of $\mathcal{C}$ as
residual solely to the front half of the denoising DiT, rather than the end
half as in U-Net ControlNet.

Formally, given a tuple of agnoistic-conditon sequence
$\left(x_{a},d_{p}\right)\in\mathbb{R}^{f\times H\times W\times 3}$ , the VAE
Encoder $\mathcal{E}$ yields the latent
$\left(z_{a},z_{p}\right)\in\mathbb{R}^{f\times h\times w\times 4}$ that are
further concatenated with the resized mask $m_{c}\in\mathbb{R}^{f\times
h\times w\times 1}$. Then the resulting latent of size $\mathbb{R}^{f\times
h\times w\times 9}$ is patchified and passed through a zero-initialized linear
layer before sending to the ID ControlNet. The output signals of $\mathcal{C}$
are directly infused into the denoising DiT as feature residuals. This setup
enables $\mathcal{C}$ to deliver precise, pixel-aligned control signals for
accurate identity preservation. To summarize:

$r_{s}=\mathcal{C}\left(\mathcal{E}\left(x_{a}\right)\odot\mathcal{E}\left(d_{p}\right)\odot
m_{c}\right),$ (7)

where $\odot$ refers to concatenation. We empirically find our ID ControlNet
is robust to accidental condition errors. Take Fig. 5(a) as an example, while
the DensePose input of the ControlNet has obvious artifacts, our VITON-DiT
still yields rational results despite inaccurate pose guidance.

###  3.5 Strategies for Long Video Generation

Figure 3: Strategies for long video generation. (a) Random agnostic condition
swap: randomly replacing agnostic images and inpainting masks with
corresponding ground-truth and all-zero masks. (b) IAR inference: generating
key-frames within each divided sequence, followed by an AR inference that
fills missing frames. Note that random swap training is the prerequisite for
the IAR inference.

Directly generating long videos is quite challenging especially when
computational resources are limited. To mitigate this, we propose a noval
random selection strategies during training and an Interpolated Auto-
Regressive (IAR) technique at inference, as depicted in Fig. 3.

####  3.5.1 Random Selection Training Strategies

Overall, these strategies include a _random agnostic condition swap_ for ID
ControlNet and a _random stride selection strategy_ for frame sampling.

Random agnostic condition swap: During training, we randomly sample k frames
and swap their agnostic images $x_{a}$ to corresponding ground-truth frames,
and set their agnostic mask $m_{c}$ to all-zero. More precisely, this strategy
is merely applied for the ID ControlNet while the input to the denoising DiT
remains unchanged so as to avoid model collapse. Without the random swap, it
becomes intractable to perform auto-regressive long video generation at test
time.

Random stride selection strategy: During training, we randomly vary the stride
used when sampling frames, and the range of the selected stride is determined
according to the average frame counts of clips in each respective dataset.
This increases the probability of the model seeing diverse viewpoints, such as
side views or back views. As a result, the model gains an improved ability to
generate temporally coherent try-on videos across different body views.

####  3.5.2 Interpolated Auto-Regressive (IAR) Inference.

Long video generation typically relies on an auto-regressive (AR) approach for
inference [34], but this method often leads to quality degradation of the
generated content over time. Inspired by the divide-and-conquer algorithm
[45], we introduce a new IAR technique that boosts the traditional auto-
regressive approach for generating high-quality extended videos. It bifurcates
the video generation into two sub-tasks: key-frame generation and frame
filling. Specifically, for generating a $f$-frame video, the IAR first divides
it into $n$ sub-videos, predicts the starting frames within each sub-video
based on the provided conditions, and subsequently conducts AR generation to
fill the missing $f-n$ frames. The way of AR is to iteratively set the last
$j$ frames of the previous video clip as the condition to generate the current
clip. The key is at each iteration, we will replace the agnostic condition
with the aforementioned $n$ interpolated frames if available. This prevents
the model from suffering quality degradation due to occlusions, and also
benefits from the AR that ensures smooth video footage, as evidenced in Fig.
6.(c) and Table. 4.

##  4 Experiments

###  4.1 Implementation Details

Dataset Construction. We collect an unpaired dataset of human dance videos
that encompasses a wide variety of clothing, backgrounds and body motions. We
segment the collected thousands of videos using scene detection tool
222https://github.com/Breakthrough/PySceneDetect and filer out clips with
multiple persons or a small portion of people, resulting in over 15,000 high-
quality video clips. We further incorporate the FashionVideo [46] and the
Tiktok dataset [20] for training. We deliberately select 50 clips with diverse
identities and backgrounds to serve as a new benchmark for evaluating video
try-on outcomes.

Multi-Stage Self-Supervised Training. To exploit the unpaired dataset, we
progressively train the model in a self-supervised manner. We segment out the
clothing in a random frame of each video clip by human parsing [11], and
augment it with random rotation and resizng before sending to the garment
extractor. We first load pretrained weights of OpenSora, and arrange the
training of the three modules in the following order:

  * •

_Image pretraining for garment extractor_ : We only train the garment
extractor and freeze all others to reconstruct the person image from its
parsed clothing. The motivation is to expose the model to a larger volume of
clothing imagery, while also reinforcing its ability to generate human images.

  * •

_Image pretraining for ID ControlNet_ : We now incorporate the ID ControlNet
and set all parameters trainable except for the SSA of the denoising ST-DiT.
The training objective is same as the first stage.

  * •

_Video fine-tuning for VITON-DiT_ : Lastly, we open all parameters for
training except the SSA modules of the denoising ST-DiT.

Hyper-parameters Setting. We train VITON-DiT with two resolutions of 192x256
and 384x512. We use the low-resolution version for qualitative and
quantitative comparison with baselines on the standard VVT dataset [6] and for
the ablation study. And the high-resolution model is for demo purposes. We
adopt the AdamW optimizer [27] with a fixed learning rate of 1e-5. The models
are trained on 8 A100 GPUs. In the first two training stages, we utilized
unpaired image data extracted from videos, and merge them with the existing
VITON-HD dataset [5]. We set $f=36$ frames for low-resolution model and $f=16$
for the high-resolution model. We keep the same number of ST-DiT blocks $N=28$
with OpenSora, and set the patch size $p$ equal to 2, the hidden dimension
$d=1152$.

Figure 4: Qualitative comparison with baselines. Our VITON-DiT outperforms
other baselines in terms of consistent preservation of garment shape and
color, as well as stable clothing-person alignment cross varing camera
distances.

###  4.2 Qualitative Results

Fig.4 presents the visual comparison between VITON-DiT and other baselines on
the VVT dataset. It is clear that GAN-based methods like PBAFN [10] (Fig.
4(a)) and ClothFormer [21] (Fig. 4(d)), are prone to clothing-person
misalignment due to the inaccurate garment warping procedure. This becomes
more severe for PBAFN on videos containing varied sizes of individuals.
Although ClothFormer can handle smaller proportions of people, the generated
images are often blurry and exhibit distorted cloth texture. The diffusion-
based StableVITON [22] produces relatively accurate single-frame results.
However, due to the image-based training, StableVITON does not account for
temporal coherence, resulting in noticeable jitters between consecutive frames
(Fig. 4(c)). Tunnel Try-on [43] in Fig. 4(e) is a concurrent work that adapts
U-Net diffusion model to video try-on. Despite reasonable results, the
generated clothes exhibit obvious color discrepancy compared with the input
ground truth. Additionally, due to the supervised setting, it is constrained
to handling product clothing images, and the provided videos on its website
are only a few seconds long.

In contrast, our VITON-DiT seamlessly integrates the scalable DiT and is
trained self-supervisedly, allowing for accurate single-frame try-on with high
inter-frame consistency. As depicted in Fig. 4(f), the letters on the chest of
the clothing adhere to the input shape and color, and are correctly positioned
as the subject moves closer to the camera. Furthermore, we provide additional
qualitative results using our unpaired dataset to demonstrate the robust try-
on capabilities and practicality of our VITON-DiT. Fig. 1 showcases various
results generated by VITON-DiT, including scenarios involving complex dance
motions against real-world backgrounds. By integrating attention fusion, our
method effectively adapts to different types of human movements and clothing,
resulting in high-detail preservation and temporal consistency in the
generated try-on sequences.

Table 1: Quantitative comparison on VVT dataset. The best results are denoted as Bold. Method | SSIM$\uparrow$ | LPIPS$\downarrow$ | $VFID_{I3D}\downarrow$ | $VFID_{ResNeXt}\downarrow$
---|---|---|---|---
CP-VTON [36] | 0.459 | 0.535 | 6.361 | 12.10
FW-GAN [6] | 0.675 | 0.283 | 8.019 | 12.15
PBAFN [10] | 0.870 | 0.157 | 4.516 | 8.690
ClothFormer [21] | 0.921 | 0.081 | 3.967 | 5.048
AnyDoor [4] | 0.800 | 0.127 | 4.535 | 5.990
StableVITON [22] | 0.876 | 0.076 | 4.021 | 5.076
Tunnel Try-on [43] | 0.913 | 0.054 | 3.345 | 4.614
VITON-DiT (ours) | 0.896 | 0.080 | 2.498 | 0.187333Owing to the absence of open-source codes for VFID metrics, we provide our implementation at https://github.com/ZhengJun-AI/vfid-metrics for fair comparison.
Table 2: Ablation study for data scales and training strategies. Data-F | Data-S | TSA | SSA($\mathcal{C}$) | SSA($\mathcal{G}$) | SSIM$\uparrow$ | LPIPS $\downarrow$
---|---|---|---|---|---|---
✓ |  | ✓ | ✓ | ✓ | 0.702 | 0.259
| ✓ | ✓ |  |  | 0.681 | 0.287
| ✓ | ✓ | ✓ |  | 0.707 | 0.257
| ✓ | ✓ | ✓ | ✓ | 0.723 | 0.240

###  4.3 Quantitative Results

The quantitative results are reported in Tab. 1. For the single-frame
evaluation, we adopt the Structural Similarity Index (SSIM) [39] and the
Learned Perceptual Image Patch Similarity (LPIPS) [48] as the evaluation
metrics. To assess the video-based performance, we employ the Video Fréchet
Inception Distance (VFID) [6] that utilizes 3D convolution to evaluate both
the visual quality and temporal consistency of the generated results. We adopt
two CNN feature extractors for VFID, including the I3D [2] and the
3D-ResNeXt101 [15].

As our training data contains a large volume of unpaired images, i.e., the
clothing is parsed from the person images and may manifest defects like
missing holes in Fig. 1, this increases the model’s imagination ability yet
inevitably leads to performance drop on the VVT dataset solely composed of
product garment images. While the other baselines are all trained on paired
dataset similar to VVT, and thus outperform the SSIM and the LPIPS image
metrics. Despite inferior single-frame scores on the product clothing images,
we beat the other methods on the more important video metrics, highlighting
the advantage of our data utilization and training strategies in generating
spatio-temporal consistent videos.

Table 3: Ablation study for Garment Extractor $\mathcal{G}$ and ID ControlNet $\mathcal{C}$. SCA($\mathcal{P}$) | SCA($\mathcal{C}$) | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
$\times$ | $\times$ | 0.8520 | 0.1168
✓ | $\times$ | 0.8584 | 0.1086
✓ | ✓ | 0.8617 | 0.1050

###  4.4 Ablation Study

We conduct extensive ablations for VITON-DiT on our collected dataset and the
standard VITON-HD image dataset [5] to verify the effectiveness of each
components. Note that all ablation results are produced with the resolution of
192x256.

Figure 5: Ablation study on the quantity of data. It is clear that as the
quality and quantity of data increases, the model’s visual performance also
gradually improves accordingly.

####  4.4.1 Data Scaling

To study the impact of data scale, we construct four variants of training
datasets from the collected videos, including the Flawed (Data-F), Small
(Data-S), Medium (Data-M) and Large (Data-L) forms. As shown in Fig. 5(d), we
denote the combination of VVT [6], FashionVideo [46] and the Tiktok dataset
[20] as "Others", while refer "HQ Data" to filtered high-quality videos clips
with proportion of human body exceeding $30\%$ and "LQ Data" to the low-
quality unfiltered clips.

As evident by Table. 2 and Fig. 5, LQ Data heavily harms the model due to the
existence of extremely small or invalid persons in frames. While the model
trained with even less yet high-quality data (i.e., Data-S) significantly
outperforms Data-F. Moreover, a greater quantity of HQ data gains a better
human body prior, allowing the model to produce reasonable results even with
inaccurate pose guidance, as depicted in Fig. 5 (Data-M and Data-L columns).

Figure 6: Ablation study on: (a) the train strategy for video fine-tuning ,
(b) the effectiveness of attention fusion and (c) the Interpolated Auto-
Regressive (IAR) technique.

####  4.4.2 Training Strategy for Video Fine-tuning

We also ablate full-tuning and freeze-tuning for the 3rd training stage. The
setups are:

  1. 1.

setting TSA modules of both the denoising ST-DiT $\mathcal{P}$ and the ID
ControlNet $\mathcal{C}$ trainable, and freeze all other parameters;

  2. 2.

setting TSA and SSA modules of the ID ControlNet $\mathcal{C}$ trainable;

  3. 3.

setting TSA and SSA modules of both the ID ControlNet $\mathcal{C}$ and the
garment extractor $\mathcal{G}$ trainable (i.e., full-tuning of $\mathcal{C}$
and $\mathcal{G}$).

From Tab. 2 and Fig. 6(a), it is clear that full-tuning is preferred to
freeze-tuning, yielding more faithful results both qualitatively and
quantitatively.

####  4.4.3 Effectiveness of Attention Fusion

We drop the Spatial Cross-Attention (SCA) in the ID ControlNet $\mathcal{C}$
and train the model both in freeze-tuning and full-tuning manners: (1) only
train $\mathcal{C}$ and $\mathcal{G}$ while freeze SCA of the denoising DiT
and (2) train all parameters except the SSA of the denoising DiT. Moreover,
our full model is also added for comparison.

Since the attention fusion mainly affects the spatial dimension, we perform
this ablation right after the 2nd training stage (i.e., without TSA loaded) on
the VITON-HD image dataset. As indicated in Tab. 3 and Fig. 6(b), our full
model with SCA in $\mathcal{C}$ is more capable of recovering the garment
texuture compared to other variants.

Table 4: Ablation study for IAR. Method | $VFID_{I3D}$ | $VFID_{ResNeXt}$
---|---|---
AR | 3.899 | 6.486
IAR | 3.674 | 5.902

####  4.4.4 Effectiveness of IAR

Fig. 6.(c) illustrates our proposed Interpolated Auto-Regressive (IAR)
inference compared to the vanilla AR approach. Our newly proposed IAR
technique enables more robust handling of occlusions and better recovery of
texture details. The metrics reported in Tab. 4 (on our benchmark) indicate
the superiority of our IAR method.

##  5 Conclusions

We propose the first DiT-based video try-on network (VITON-DiT) for
facilitating scalable unpaired video virtual try-on. By utilizing an attention
fusion process as well as a novel random selection and IAR strategies, VITON-
DiT is able to faithfully recover clothing details in the generated videos
spanning tens of seconds. Experiments highlights VITON-DiT’s ability to handle
diverse clothing and complex body movements, outperforming previous methods in
term of spatio-temporal consistency under real-world scenarios. We believe
this work will pave the way for new scalable approaches, enabling the
utilization of the vast amount of readily available unlabeled video data.

## References

  * [1] A. Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22563–22575, 2023.
  * [2] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017.
  * [3] Weifeng Chen, Tao Gu, Yuhao Xu, and Chengcai Chen.  Magic clothing: Controllable garment-driven image synthesis.  arXiv preprint arXiv:2404.09512, 2024.
  * [4] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.  Anydoor: Zero-shot object-level image customization.  arXiv preprint arXiv:2307.09481, 2023.
  * [5] Seung-Hwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo.  Viton-hd: High-resolution virtual try-on via misalignment-aware normalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14131–14140, 2021.
  * [6] Haoye Dong, Xiaodan Liang, Xiaohui Shen, Bowen Wu, Bing-Cheng Chen, and Jian Yin.  Fw-gan: Flow-navigated warping gan for video virtual try-on.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019.
  * [7] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2758–2766, 2015.
  * [8] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach.  Scaling rectified flow transformers for high-resolution image synthesis, 2024.
  * [9] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2021.
  * [10] Yuying Ge, Yibing Song, Ruimao Zhang, Chongjian Ge, Wei Liu, and Ping Luo.  Parser-free virtual try-on via distilling appearance flows.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8485–8493, 2021.
  * [11] Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin.  Instance-level human parsing via part grouping network.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision – ECCV 2018, pages 805–822, Cham, 2018. Springer International Publishing.
  * [12] Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai.  Sparsectrl: Adding sparse controls to text-to-video diffusion models.  arXiv preprint arXiv:2311.16933, 2023.
  * [13] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  International Conference on Learning Representations, 2024.
  * [14] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S. Davis.  Viton: An image-based virtual try-on network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7543–7552, 2018.
  * [15] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018.
  * [16] Sen He, Yi-Zhe Song, and Tao Xiang.  Style-based global appearance flow for virtual try-on.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3470–3479, 2022.
  * [17] hpcaitech.  Open-sora: Democratizing efficient video production for all.  https://github.com/hpcaitech/Open-Sora, 2024.
  * [18] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo.  Animate anyone: Consistent and controllable image-to-video synthesis for character animation.  arXiv preprint arXiv:2311.17117, 2023.
  * [19] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  * [20] Yasamin Jafarian and Hyun Soo Park.  Learning high fidelity depths of dressed humans by watching social media dance videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12753–12762, June 2021.
  * [21] Jianbin Jiang, Tan Wang, He Yan, and Junhui Liu.  Clothformer: Taming video virtual try-on in all module.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [22] Jeongho Kim, Gyojung Gu, Minho Park, Sunghyun Park, and Jaegul Choo.  Stableviton: Learning semantic correspondence with latent diffusion model for virtual try-on.  arXiv preprint arxiv:2312.01725, 2023.
  * [23] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  CoRR, abs/1312.6114, 2013.
  * [24] Gaurav Kuppa, Andrew Jong, Xin Liu, Ziwei Liu, and Teng-Sheng Moh.  Shineon: Illuminating design choices for practical video-based virtual clothing try-on.  In 2021 IEEE Winter Conference on Applications of Computer Vision Workshops (WACVW), Jan 2021.
  * [25] PKU-Yuan Lab and Tuzhan AI etc.  Open-sora-plan, April 2024.
  * [26] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1096–1104, 2016.
  * [27] Ilya Loshchilov and Frank Hutter.  Fixing weight decay regularization in adam.  ArXiv, abs/1711.05101, 2017.
  * [28] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [29] OpenAI.  "sora: Creating video from text.".  https://openai.com/sora, 2024.
  * [30] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  arXiv preprint arXiv:2212.09748, 2022.
  * [31] Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21:140:1–140:67, 2019.
  * [32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer.  High-resolution image synthesis with latent diffusion models.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022.
  * [33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham, 2015. Springer International Publishing.
  * [34] Jonathan Tseng, Rodrigo Castellon, and C Karen Liu.  Edge: Editable dance generation from music.  arXiv preprint arXiv:2211.10658, 2022.
  * [35] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Neural Information Processing Systems, 2017.
  * [36] Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, and Liang Lin.  Toward characteristic-preserving image-based virtual try-on network.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 589–604, 2018.
  * [37] Rui Wang, Hailong Guo, Jiaming Liu, Huaxia Li, Haibo Zhao, Xu Tang, Yao Hu, Hao Tang, and Peipei Li.  Stablegarment: Garment-centric generation via stable diffusion, 2024.
  * [38] Xiang* Wang, Hangjie* Yuan, Shiwei* Zhang, Dayou* Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  2023\.
  * [39] Zhou Wang, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  In IEEE Transactions on Image Processing, volume 13, pages 600–612, 2004.
  * [40] Zhenyu Xie, Zaiyu Huang, Fuwei Zhao, Haoye Dong, Michael Kampffmeyer, and Xiaodan Liang.  Towards scalable unpaired virtual try-on via patch-routed spatially-adaptive gan.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 2598–2610. Curran Associates, Inc., 2021.
  * [41] Zhenyu Xie, Zaiyu Huang, Fuwei Zhao, Haoye Dong, Michael C. Kampffmeyer, and Xiaodan Liang.  Towards scalable unpaired virtual try-on via patch-routed spatially-adaptive gan.  In Neural Information Processing Systems, 2021.
  * [42] Yuhao Xu, Tao Gu, Weifeng Chen, and Chengcai Chen.  Ootdiffusion: Outfitting fusion based latent diffusion for controllable virtual try-on.  arXiv preprint arXiv:2403.01779, 2024.
  * [43] Zhengze Xu, Mengting Chen, Zhao Wang, Linyu Xing, Zhonghua Zhai, Nong Sang, Jinsong Lan, Shuai Xiao, and Changxin Gao.  Tunnel try-on: Excavating spatial-temporal tunnels for high-quality virtual try-on in videos.  arXiv preprint, 2024.
  * [44] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen.  Paint by example: Exemplar-based image editing with diffusion models.  arXiv preprint arXiv:2211.13227, 2022.
  * [45] Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming, Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan.  Nuwa-xl: Diffusion over diffusion for extremely long video generation, 2023.
  * [46] Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal.  Dwnet: Dense warp-based network for pose-guided human video generation.  ArXiv, abs/1910.09139, 2019.
  * [47] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  * [48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  * [49] Shiwei* Zhang, Jiayu* Wang, Yingya* Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qing, Xiang Wang, Deli Zhao, and Jingren Zhou.  I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models.  2023\.
  * [50] Xie Zhenyu, Huang Zaiyu, Dong Xin, Zhao Fuwei, Dong Haoye, Zhang Xijin, Zhu Feida, and Liang Xiaodan.  Gp-vton: Towards general purpose virtual try-on via collaborative local-flow global-parsing learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
  * [51] Xiaojing Zhong, Zhonghua Wu, Taizhe Tan, Guosheng Lin, and Qingyao Wu.  Mv-ton: Memory-based video virtual try-on network.  In Proceedings of the 29th ACM International Conference on Multimedia, Oct 2021.