# Significance of Skeleton-based Features in Virtual Try-On

Debapriya Roy, Sanchayan Santra, Diganta Mukherjee, and Bhabatosh Chanda  D.
Roy is with the dept. of Institute of Advancing Intelligence at the TCG
Centres for Research and Education in Science and Technology, Kolkata, India.
E-mail<EMAIL_ADDRESS>Dr. S. Santra is with the Institute for
Datability Science, Osaka University, Osaka, Japan.
E-mail<EMAIL_ADDRESS>Prof. Mukherjee is with the Indian
Statistical Institute, kolkata, India.
E-mail<EMAIL_ADDRESS>Prof. Chanda was formerly with the Indian
Statistical Institute, kolkata, India and currently he is with Indian
Institute of Information Technology Kalyani, India .
E-mail<EMAIL_ADDRESS>
###### Abstract

The idea of Virtual Try-ON (VTON) benefits e-retailing by giving an user the
convenience of trying a clothing at the comfort of their home. In general,
most of the existing VTON methods produce inconsistent results when a person
posing with his arms folded i.e., bent or crossed, wants to try an outfit. The
problem becomes severe in the case of long-sleeved outfits. As then, for
crossed arm postures, overlap among different clothing parts might happen. The
existing approaches, especially the warping-based methods employing Thin Plate
Spline (TPS) transform can not tackle such cases. To this end, we attempt a
solution approach where the clothing from the source person is segmented into
semantically meaningful parts and each part is warped independently to the
shape of the person. To address the bending issue, we employ hand-crafted
geometric features consistent with human body geometry for warping the source
outfit. In addition, we propose two learning-based modules: a synthesizer
network and a mask prediction network. All these together attempt to produce a
photo-realistic, pose-robust VTON solution without requiring any paired
training data. Comparison with some of the benchmark methods clearly
establishes the effectiveness of the approach.

###### Index Terms:

Virtual Try-on, Thin plate spline transformation, deep neural networks, hand-
crafted geometric features.

##  I Introduction

Figure 1: Results of different methods for different input clothing, model,
and person combinations. A visual comparison with some benchmark methods
illustrates the efficacy of the proposed method. See

The online share of total global retail sales has shown a steady increase over
the last few years. The fashion industry with a value of around 3 trillion US
dollars is one of the major contributors in online retail sales
111https://fashinnovation.nyc/fashion-industry-statistics. E-retailers are
constantly focusing to improve consumer’s shopping experience. However, buying
clothing items online always includes the uncertainty of fitting and
appearance. Virtual try-on (VTON) solves this problem to some extent by giving
a consumer the provision to try a clothing item at the comfort of their home.
Besides, for the consumers, reluctant to try a product in-store, VTON may be a
great time efficient alternative solution to them.

The 3D VTON methods [1, 2, 3, 4] provides visually pleasing VTON experiences
but requires 3D human body parametric models [5] or 3D scanned datasets.
Compared to this image-based VTON approaches [6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17] require only two images, the garment image, and the reference
person image. Most of the existing VTON methods adopt a geometric deformation-
based approach. They estimates a non-linear transformation, usually the thin
plate spline transformation (TPS), to deform the source clothing to fit the
person. This is then combined with the person image to generate the final try-
on image. Some VTON method [6, 11, 10, 15] employed a Geometric Matching
Network (GMN) which is a Deep Neural Network (DNN) that predicts the TPS
transform directly from the input images by leveraging the correlation between
the features extracted from these images. A landmark-based image registration
[18] approach to compute the TPS transformation was proposed by LGVTON [8]. It
computes the transform using some explicitly specified source to target
control point correspondences. However, both types of methods often produce
inconsistent results when the person is posing with significant arm bending
(see Fig. 2). This mainly occurs due to the smoothness constraint of TPS which
restricts bending in the predicted warp. In addition, the formulation of TPS
models some rearrangements of control points in the image plane; therefore,
not intended for applications requiring modeling of overlap or fold [19] in
different parts of the target warp.

Most of the approaches do not consider occlusion and pose variability in the
source clothing, as they consider a separate clean garment image (see the 1st,
3rd column of Fig. 1) as the source clothing. We call such methods Cloth-To-
Person (C2P) methods [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20]. These
methods perform poorly when the source clothing is in the form of a model
wearing it. For instance, see the difference in the performance of ACGPN and
He et al. [20] with different clothing sources in Fig. 1. Note that the
methods taking the model image as the clothing source are named as Model-To-
Person (M2P) methods. M2E-TON [21] and Liu et al. [22] used the idea of body
shape correspondences in M2P VTON. Compared to M2E-TON, Liu et al.’s solution
can handle cases of large pose variability between model and the person and
also texture-variability of clothes. However, in case of complex textures on
the clothing e.g., stripes, floral, etc., it falls short in preserving
details.

In addition to the above-mentioned approaches, the problem of VTON is also
explored using GAN-based [23] attribute manipulation approaches [24, 25, 26,
27]. While these methods often meet the standards of photo-realism but still
fail to retain the exact texture patterns of the source clothing in the try-on
output and are also limited to simple poses.

Other than the above-discussed approaches another kind is the appearance flow-
based approach [28, 29, 30, 20]. Such methods find the cloth flow from the
source to the target images. However, some of this methods [28, 29, 30] often
fall short when there is a large misalignment between the garment and the
person’s body parts. An attempt to overcome this, by capturing the global
context, is proposed in [20], but, in case of significant arm bending it shows
poor performance (see Fig. 1).

Considering all these, we attempt to achieve a pose robust M2P VTON solution
without requiring any paired training data, i.e., images of the same person
with and without wearing the model’s clothing. Unlike previous approaches we
divide the source clothing into parts i.e., sleeves and torso, followed by
warping each of these parts separately, before finally combining them. This
approach attempts to solve the overlap issue. Moreover, it opens the
flexibility of employing clothing part specific transformation. Considering
the limitations of TPS we employ this for warping only that part of the
clothing which is expected to undergo limited deformation, i.e., torso.
However, the human arms can move in various ways causing different
deformations in the sleeves most of which can not be modeled by TPS transform
due to its bending constraint.

We leverage the idea of landmark registration in the current problem context
[8] due to its implicit consideration of human anatomical constraints; which
controls undesirable deformation in the target warp. However, considering only
landmark correspondences in warping the sleeves is difficult. So, inspired
from field transform [31] we propose a hand-crafted feature-based warping
technique that is consistent with human arm movements. Instead of landmark
correspondences, here we consider the correspondences of the straight line
segments between the consecutive landmarks. This imposes an additional
constraint that is well justified with human anatomy (see Fig. 4(c)).

Figure 2: Demonstration of simple and complex human poses.

M2P VTON methods face the inherent challenge of occlusion in the source
clothing (see the clothing area near the neck occluded by the model’s hair in
2nd row of Fig. 2). We propose a mask prediction network (MPN) to tackle this.
The output of MPN aids in distinguishing the areas of the target clothing
occluded in the source. The proposed image synthesizer network (ISN)
interpolates these occluded regions and also produces a seamless try-on image.
MPN also adds to the faster computation of the target warp.

To summarize, we make the following contributions, (1) We propose a self-
supervised model-to-person VTON solution that shows significant performance
improvement over the previous methods, especially for complex human arm poses
(see Fig. 2). (2) The proposed part-by-part warping technique works well in
computing the target warps requiring overlap among different parts of it. (3)
We discuss the limitations of TPS transform in the current problem context and
propose a hand-crafted feature based warping method that is consistent with
the human body geometry. (4) We propose a mask prediction network (MPN) that
aids in identifying the occluded regions of the target clothing and thereby
guides our proposed image synthesizer network (ISN) to fill respective
clothing details in those regions.

Note that this work focuses on upper body clothing only; however, can be
extended for lower body clothing too. In the rest of the paper, we present a
brief literature survey in Sec. II, solution approach in Sec. III, and, the
experiments in Sec. IV. Finally, we conclude in Sec. V. Code will be uploaded
after acceptance.

##  II Related Works

Image-based virtual try-on approaches can be categorized into warping-based
and appearance flow based approaches. Among the warping-based approaches [7,
6, 10, 8, 9, 12, 15, 11] the popular warping function is Thin Plate Spline
transformation function.

The deep neural network related warping-based approaches [6, 10, 8, 9, 12, 15,
11] learns the features from the clothing and the reference person images and
correlates them. The features learnt from that correlation map is used to
predict the parameters of TPS. The network employing this is called geometric
matching network (GMN). CP-VTON [6] first employed GMN and showed improvement
over shape context warp idea of VITON [7]. VTNFP [10] additionally
incorporates non-local mechanism [12] in the feature extraction part of the
GMN. However, high flexibility of TPS, often causes GMN to produce undesirable
warping results in presence of complex patterns in clothes. ACGPN [15]
proposed to employ a second-order difference constraint to control the grid
deformations causing undesirable warps. LGVTON [8] proposed a landmark guided
VTON approach which used landmark based image registration approach to
estimate $\theta_{TPS}$. Roy et al. [9] employs GMN but learns the
correspondences of features of the densepose representations [32] of the model
and the person instead of directly learning from the images. M2E-TON [21] also
used densepose representations but used the idea of barycentric coordinate
based texture transfer from model to person. Some GAN based image synthesis
approach for attribute manipulation is proposed in [25, 26, 27]. Recently,
Lewis et al. [27] proposed a styleGAN based novel approach of try-on on
unpaired training data. For a given model and person pair it finds the optimal
interpolation coefficients per layer to generate a try-on output. GarmentGAN
[16] employs two separate GANs for shape generation and appearance generation
to predict the final try-on output. SieveNet[17] introduces a duelling triplet
loss to improve the quality of the try-on output. PASTA-GAN [33] proposed a
patch-routed disentanglement module that disentangles style and spatial
information in a garment. It’s key contribution is employing source and target
person’s pose-based keypoints to decouple a garment into normalized patches
and then reconstructing them to the warped garment. While we also employ pose
key points but the idea of our geometric approach of warping is quite
different from their approach.

Other than warping, appearance flow based approaches are also proposed [28,
29, 30, 20, 34]. The very first work in this direction is proposed in [30],
which predicts the clothing flow from the source to target images. The main
idea of appearance flow is to predict the sampling grid for clothes warping. A
styleGAN [35] based appearance flow estimation method is proposed in [20]
which uses the style vector to capture the global context of the image. This
attempts to overcomes the challenges due to considering only local appearance
flow [28, 29, 30].

Summarizing the contributions of the above methods, we see the above methods
has contributed to improve the photo realism of the output. Also some method
shows ability to handle complex hand postures. But most of the methods
struggle in case of warping the sleeves of the long-sleeved clothing in case
of significant bending of arms. The contribution of this paper targets this
limitation of others. Our method works on variety of human poses, irrespective
of human body shape, outfit shapes and also attempts to retain the exact
texture and colors of the source clothing in the target output.

Figure 3: Illustration of our overall virtual try-on approach.

##  III Approach

###  III-A Overview

Given a model’s image $M$ and a person’s image $P$, we propose, the problem of
virtual try-on (VTON) as, $P^{\prime}=\mathcal{F}(M,P)$, where $P^{\prime}$
refers to the image of the person wearing the reference clothing of $M$. We
compute $P^{\prime}$ in mainly three steps. First, We employ a mask predictor
network (MPN) (Sec. III-B) that predicts the target clothing mask, denoting
the trialed clothing region of the expected try-on output. Second, we extract
the model’s outfit $c$ using a human parsing method [36]222Human parsing is
the task of segmenting a human image into different fine-grained semantic
parts e.g., head, torso, arms, legs, upper-clothes, etc. and compute the
target warp $c^{\prime}$ that fits the person $P$, using our proposed warping
method (Sec. III-C). Third, we propose a novel synthesis step employing an
image synthesizer network (ISN) that seamlessly combines $c^{\prime}$ with $P$
to generate the final try-on output $P^{\prime}$ (Sec. III-D). Below we
discuss all the steps in detail. A simplistic block diagram demonstrating the
different steps of our approach is shown in Fig. 3.

###  III-B Predicting the target clothing mask

This step predicts the target clothing mask serving two purposes in the
subsequent stages of this work: (1) it aids the synthesis module to identify
the occluded regions of the source clothing which needs to be interpolated to
produce a seamless try-on output, (2) it helps our proposed warping method in
reducing the time of computation of the target warp corresponding to the
sleeves of the source clothing. This will be elaborated on later during the
discussion of the warping method.

To predict the mask, we train a convolutional neural network (CNN), named the
mask prediction network (MPN), with the following inputs - (1) the
underclothing body shape of the model and the person, encoded using their
densepose representations [32] 333A densepose representation maps all human
pixels of an RGB image, to the 3D surface of the human body, thus providing a
precise estimate of the human body shape under the clothing.. In addition, we
also provide the face and head segments of both the model and the person
extracted using [36]. (2) the mask of $c$, and (3) semantically segmented
human parts [37] of the model. A block diagram of MPN along with the inputs
and output are demonstrated in Fig. 4(b). Note that providing the face and
head segments along with the densepose as input gives the whole detail of a
human body. Whereas, a more refined level of details is provided by the
segmented human parts.

The architecture of MPN is demonstrated in Fig. 4(b). We incorporate
correlation layers in MPN that computes the linear relationship between the
body shape and pose features of the model and that of the person (Observe Fig.
4(b)). The human parsing branch at the end of this module (demonstrated in
Fig. 4(b)) predicts the human parsing of $P^{\prime}$ given the non-target
clothing details of $P$ e.g., the lower body clothing, the face and hair
segments, and the predicted mask from the previous branch. This is enforcing
another constrain on the main branch to predict a mask consistent with the
final human parsing.

###  III-C Predicting the target warp

Figure 4: (a) Illustration of the method for predicting the target warp, (b)
Block diagram of the proposed mask predictor network (MPN), (c) Demonstration
of human landmarks, (d) Demonstration of human part parsing [37] results and
pose key points.Best viewed in electronic version.

Figure 5: (a) (top)Elbow flexion and extension (picture courtesy [38]),
(bottom) stretch and folds in clothing sleeve due to elbow flexion i.e., arm
bending (picture courtesy [39]). (b) A graphical illustration of the arm
bending phenomenon in human. A sample pair of model and person relevant to the
illustrated scenario is given above for better understanding, (c) Plot of
functions $f(\phi_{1},\phi_{2})$, $g(\phi)$ and $h(\phi_{1},\phi_{2})$, (d)
Geometrical illustration of our warping method for sleeves warping for the two
different scenarios. (Left) Example of a case when assumption 1 holds. (Right)
Example of a case when assumption 2 holds. Here $\\{A,B,C\\}$ and
$\\{A^{\prime},B^{\prime},C^{\prime}\\}$ are the landmarks corresponding to
the arm of the person and the model respectively. $X$ refers to the point
belonging to the sleeve segment of the target warp and $X^{\prime}$ is its
corresponding source pixel.

This step predicts $c^{\prime}$ from $c$. We do it part-by-part, by first
segmenting $c$ into 3 semantic parts - torso ($c_{torso}$), left sleeve
($c_{lsleeve}$), right sleeve ($c_{rsleeve}$) and thereafter opting skeletal
structure specific warping approaches. Different parts of human body have
different skeletal structures causing different limitations of movement in
each part. This effects the transformation of the related clothing part when
the clothing is transferred from one human’s shape-pose state to another. For
instance, in our case from the model’s shape-pose state to that of the
person’s. Our idea of part-by-part warping is inspired from this fact only.
Below, first we discuss our proposed method of warping the sleeves i.e.,
predicting $c_{lsleeve}^{\prime}$, $c_{rsleeve}^{\prime}$, which is one of the
major contributions of this paper. Thereafter, we discuss the approach of
warping the torso part of the clothing i.e., computing $c_{torso}^{\prime}$.
The human part segmentation method [37] used to segment the clothing into
different parts is portrayed in Fig. 4(d).

In order to compute the target warp of the model’s sleeves we employ feature
correspondences from the arm of the model to that of the person. Considering
features related to the arm is obvious here because sleeves relate to arm part
only. Formally, we propose geometric warping approach, where for each source
pixel $X^{\prime}\in c_{lsleeve}(c_{rsleeve})$ our goal is to compute the
corresponding target pixel $X\in c_{lsleeve}^{\prime}(c_{rsleeve}^{\prime})$.

In human skeleton each arm consists of 3 bones, humerus bone in the upper arm,
and radius and ulna in the lower arm. Now considering radius and ulna
together, an arm may be represented as consisting of 2 bones, each
corresponding to the upper and lower arm connected at a the common elbow
joint. Using human pose representation [40] we can get the location of this
elbow joint and the line segments corresponding to the two bones. For instance
as shown in Fig. 4 (c) each arm contains only 3 landmarks. The line segments
connecting these landmarks for example, {(2,3), (3,4)} for right arm and
{(5,6), (6, 7)} for left arm may be considered the representative of the
bones. Hence, geometrically, consider the arm of the target person consists of
the following line segments, {($B$, $A$), ($B$, $C$)} and that of the source
person as {($B^{\prime}$, $A^{\prime}$), ($B^{\prime}$, $C^{\prime}$)}. Please
see Fig. 5{(b), (c)} for visual clarifications. We employ the correspondences
of these lines, for instance from ($B^{\prime}$, $A^{\prime}$) to ($B$, $A$)
and ($B^{\prime}$, $C^{\prime}$) to ($B$, $C$) for computing the target sleeve
warp. Note that instead of only landmarks or bone joints correspondences i.e.,
{A’, B’, C’} to {A, B, C} as proposed in [8]444LGVTON considers
correspondences of 3 landmarks of each arm from source to target. our approach
is more constrained as we are considering the correspondences of the human
bones which is anatomically more consistent.

We use polar coordinates relative to the line segments $BA$ and
$B^{\prime}A^{\prime}$ to define the location of $X$ and $X^{\prime}$
respectively555Note that considering the locations with respect to the lines
$BC$ and $B^{\prime}C^{\prime}$ will not effect the idea of this method. Also
polar system is useful in our case, which will becomes clear with further
readings. As shown in Fig. 5(b) with respect to $BA$ location of $X$ is
$(r,\phi_{1})$ where $r=\|BX\|$ and $\phi_{1}$ = $\angle{XBA}$. Similarly with
respect to $B^{\prime}A^{\prime}$ the location of $X^{\prime}$ is
$(r^{\prime},\phi_{1}^{\prime})$. Since we already have the locations of
landmarks $A,B,C$, and, $A^{\prime},B^{\prime},C^{\prime}$, so for each $X$ we
have the values of $(r,\phi_{1})$. Now, with regard to the backward mapping
concept for each of $X$ our goal is to compute the corresponding $X^{\prime}$
i.e., the values of $(r^{\prime},\phi_{1}^{\prime})$. Note that here we
compute $X^{\prime}$ only for the
$X\in\\{c_{sleeve}^{\prime}\cap\mathcal{S}\\}$, where $\mathcal{S}$ is the
target clothing mask predicted by MPN, and,
$c_{sleeve}^{\prime}=(c_{lsleeve}^{\prime}\cup c_{rsleeve}^{\prime})$. This
aids in reducing the computation time.

Below, we first discuss the method to compute the angular coordinate
$\phi_{1}^{\prime}$ followed by discussing our idea of computing the radial
coordinate $r^{\prime}$. Now before continuing further, let us introduce some
additional notations. Let $\phi_{2}$ = $\angle{CBX}$, $\phi_{1}^{\prime}$ =
$\angle{X^{\prime}B^{\prime}A^{\prime}}$. $\phi_{2}^{\prime}$ =
$\angle{C^{\prime}B^{\prime}X^{\prime}}$, $\phi$ = $\angle{CBA}$ =
$\phi_{1}+\phi_{2}$.

In general human arm undergoes bending which is called elbow flexion. The
bending amount is quantified using the flexion angle ($\in[0^{\circ}\text{(no
bending)},145^{\circ}\text{(max bending)}]$)
666https://en.wikipedia.org/wiki/Elbow(ref to Fig. 5(a)). Due to bending the
sleeve undergoes folding in the inner part and stretching in the outer part
(Fig. 5(a)). We observed, due to bending the relative angular position of a
source pixel $X^{\prime}$ with its closest line remains unchanged in the
target warp. For instance, as shown in Fig. 5(b), X is closer to $BA$ than
$BC$ as $\phi_{1}<\phi_{2}$ so $\phi_{1}^{\prime}=\phi_{1}$. We include this
observation in the form of two assumptions,

  * •

Assumption 1 - when $X$ is closer to the line $BA$ i.e., when
$\phi_{1}<\phi_{2}$, the relative angular position of $X$ with respect to line
$BA$ i.e., $\phi_{1}$, will be equal to the angular position of $X^{\prime}$
with respect to line $B^{\prime}A^{\prime}$ which is $\phi_{1}^{\prime}$,
i.e., $\phi_{1}^{\prime}=\phi_{1}$.

  * •

Assumption 2 - In a similar sense, when $X$ is closer to the line $BC$ i.e.,
when $\phi_{2}<\phi_{1}$, then $\phi_{2}^{\prime}=\phi_{2}$. Note that saying
$\phi_{2}^{\prime}=\phi_{2}$ is equivalent to saying
$\phi_{1}^{\prime}=\phi^{\prime}-\phi_{2}$.

In the elbow region bending causes 2 kinds of warping of the sleeves, folding
and stretching as shown in Fig. 5(a). In the outer arm area near the elbow
region where generally $\phi_{1}$ and $\phi_{2}$ are very close, strictly
adhering to one of two assumptions does not produce realistic result. Instead,
then we compute $\phi_{1}^{\prime}$ as the weighted combinations of the values
of $\phi_{1}^{\prime}$ due to the two assumptions; where the weight is
computed by a function $f(\cdot,\cdot)$. Whereas, as $X$ moves towards the
upper and lower end of the arm either of the two assumptions can be used to
compute $\phi_{1}^{\prime}$. Based on this we compute $\phi_{1}^{\prime}$ for
the pixels in the outer arm region as,

$\phi_{1}^{\prime}=\phi_{1}(1-f(\phi_{1},\phi_{2}))+(\phi^{\prime}-\phi_{2})f(\phi_{1},\phi_{2}),$
(1)

where,

$f(\phi_{1},\phi_{2})=\frac{\phi_{1}^{2}}{\phi_{1}^{2}+\phi_{2}^{2}}$ (2)

The choice of this function is intuitive. Observe in Fig. 5(d) that the
functional value of $f$ takes smooth transition from 0 to 1 from the region
$\phi_{1}<\phi_{2}$ (criterion of assumption 1) to $\phi_{1}>\phi_{2}$
(criterion of assumption 2). This smooth transition ensures that near the
elbow in the outer arm area i.e., when $\phi_{1}$ is close to $\phi_{2}$ a
weighted a combination of the effects of both the assumptions hold for
stretching unlike the outer arm.

For the sleeve area in the inner arm region either of the two assumptions
holds. A rounded version of the function $f$ may be used in such case, because
rounding removes the effect of smooth transition which is not required here.
So for the pixels in the inner arm region we compute $\phi_{1}^{\prime}$ as,

$\phi_{1}^{\prime}=\phi_{1}(1-Round(f(\phi_{1},\phi_{2})))+(\phi^{\prime}-\phi_{2})Round(f(\phi_{1},\phi_{2})).$
(3)

To combine these two cases in one expression we need a function to distinctly
identify whether a pixel belongs to the inner arm or outer arm. Notice in the
inner arm $\phi<\pi$ and in the outer arm $\phi\geq\pi$ (Fig. 5(a)). Based on
this we define $g(\cdot,\cdot)$ as a logistic function that varies from 0 to 1
near $\pi$. The idea of taking a logistic instead of a step function here is
to make a smooth transition between the regions $\phi<\pi$ and $\phi>\pi$ so
that a realistic warping result is obtained. The functional form of $g$ is,

$g(\phi)=\frac{1}{1+e^{a(\pi-\phi)}}.$ (4)

Observe, when $\phi<\pi$ (inner arm), $g(\cdot,\cdot)\to 1$, and, when
$\phi>\pi$ (outer arm), $g(\cdot,\cdot)\to 0$. Now we know, in the inner arm
$f(\phi_{1},\phi_{2})$ holds and in the outer arm
$Round(f(\phi_{1},\phi_{2}))$. Combining these two using $g(\cdot,\cdot)$ we
get $h(\cdot,\cdot)$ as follows,

$h(\phi_{1},\phi_{2})=g(\phi)f(\phi_{1},\phi_{2})+(1-g(\phi))Round(f(\phi_{1},\phi_{2})).$
(5)

Hence, for all pixel $X$, we may computer $\phi_{1}^{\prime}$ as,

$\phi_{1}^{\prime}=\phi_{1}(1-h(\phi_{1},\phi_{2}))+(\phi^{\prime}-\phi_{2})h(\phi_{1},\phi_{2}),$
(6)

Now that, we have computed $\phi_{1}^{\prime}$, we need to compute
$r^{\prime}$ as previously mentioned. Before we compute $r^{\prime}$ let us
understand, a point closer to $B^{\prime}C^{\prime}$ (or
$B^{\prime}A^{\prime}$) should be proportionally similar in distance to $BC$
(respectively $BA$). To maintain this scaling effect we multiply $r$ with the
ratio of the source and the corresponding target line i.e.,
$\|B^{\prime}C^{\prime}\|$ and $\|BC\|$ (resp. $\|B^{\prime}A^{\prime}\|$ and
$\|BA\|$). For example, if $\|B^{\prime}A^{\prime}\|<\|BA\|$ then,
$r^{\prime}<r$, as $\frac{\|B^{\prime}A^{\prime}\|}{\|BA\|}<1$. A
demonstration of this case is given in Fig. 6. In this example, the line $BA$
(red-colored) is longer than $B^{\prime}A^{\prime}$, hence, the upper part of
the sleeve (rectangle) in the transformed image is scaled accordingly.

Based on this concept, we obtain the value of $r^{\prime}$ using the following
formula.

$r^{\prime}=r\left\\{(1-h(\phi_{1},\phi_{2}))\frac{\|B^{\prime}A^{\prime}\|}{\|BA\|}+h(\phi_{1},\phi_{2})\frac{\|B^{\prime}C^{\prime}\|}{\|BC\|}\right\\}.$
(7)

Figure 6: Our result depicting the effect of scaling of lines in our method.
As the length of the line increases from source to target the corresponding
part of the rectangle looks zoomed-in in the result.

Now, we have computed the radial and angular coordinates $r^{\prime}$ and
$\phi_{1}^{\prime}$ of $X^{\prime}$ respectively and, we already have the
coordinates of $A^{\prime}$ and $B^{\prime}$. Therefore, computing the
location of $X^{\prime}$ in Cartesian is straightforward. Hence, we do not
describe it in detail. Note that we compute the source pixels corresponding to
only the pixels of the target clothing regions predicted by the proposed MPN.

The procedure of sleeve warping is formally presented in Algorithm. 1. In
order to keep the main manuscript concise and to the point only some further
analysis on how our method handles occluded areas in the source clothing
regions is explained in the supplementary materials.

Algorithm 1 Algorithm for computing the source pixel $X^{\prime}\in
c_{lsleeve}(c_{rsleeve})$ corresponding to a location $X\in
c_{lsleeve}^{\prime}(c_{rsleeve}^{\prime})$. See Fig. 5(b) for a visual
understanding of the notations.

0:

0:

1: Compute the radial coordinate $r$ and angular coordinate $\phi_{1}$ of $X$
as $r=\|BX\|$ and $\phi_{1}=\angle{XBA}$

2: Given $A,B,C$ compute $\phi_{1},\phi_{2}$; where $\phi_{1}$ = $\angle{XBA}$
and $\phi_{2}$ = $\angle{CBX}$.

3: Compute $f(\phi_{1},\phi_{2})$ using Eq. 2, and $g(\phi)$ using Eq. 4 where
$\phi=\phi_{1}+\phi_{2}$

4: Compute $h(\phi_{1},\phi_{2})$ from $f(\cdot,\cdot)$ and $g(\cdot)$ using
Eq. 5

5: Compute the angular coordinate $\phi_{1}^{\prime}$ of $X^{\prime}$ using
Eq. 6

6: Compute the radial coordinate $r^{\prime}$ of $X^{\prime}$ using the Eq.7

7: Now we have the radial and angular coordinates i.e., $r^{\prime}$ and
$\phi_{1}^{\prime}$ of $X^{\prime}$ respectively. Therefore, computing the
location of $X^{\prime}$ in Cartesian coordinate system is straightforward as
we already have the coordinates of $A^{\prime}$ and $B^{\prime}$.

8: return $X^{\prime}$

To compute $c_{torso}^{\prime}$ from $c_{torso}$ we adopt human landmark
correspondences to compute TPS transform [8].

But in this application, since we are interested in the target clothing region
only, so, we compute the source pixels corresponding to the pixels belonging
only to the target clothing region predicted by the proposed MPN

###  III-D Try-on image synthesis

The objective of this stage is to combine $c^{\prime}$ with $P$ to generate a
plausible and photo-realistic try-on result. Our warping method computes
$c^{\prime}$ from only the visible clothing areas of the model’s clothing. But
due to the difference in pose between model and person some occluded areas of
the model’s clothing might become visible in the target. Also, long hairs
often occlude clothing areas. Considering the target clothing mask predicted
by MPN as $\mathcal{S}$, and the mask of the predicted warp $c^{\prime}$ as
$c^{\prime}_{m}$, we obtain such areas as,
$\mathcal{S}\cdot\bar{c^{\prime}_{m}}$, where $\cdot$ denotes dot product and
$\bar{(.)}$ denotes complement. The main goals of ISN are to inpaint this
region and produce a seamless combination of $c^{\prime}$ and $P$, while also
removing the previous clothing details from $P$.

To achieve the above objectives we trained an encoder-decoder based
convolutional neural network (CNN). We call it image synthesizer network
(ISN). The inputs to this network are: (i) the predicted target clothing mask
from the MPN, (ii) the body shape of the target person represented by its
densepose representation, (iii) instead of providing the predicted target warp
and the person image separately, we provide a combined representation of them.
To compute this, the upper body details i.e., previous clothing (e.g.,
t-shirt, etc.), exposed skin areas are masked in the person image, and the
predicted warp is overlaid on it. (iv) The inpainting mask
$\bar{\mathcal{S}}\cdot\bar{c^{\prime}_{m}}$ (0 denotes the pixel with missing
details).

We trained this network using self-supervision employing inpainting related
loss functions as suggested in [41].

##  IV Experiments

Dataset. We experiment on MPV dataset [11] which is a benchmark dataset in
current problem domain. It contains 35,687 images of fashion models captured
from various angles with their upper-body clothing images given separately
that in total is 13,524. We randomly collected a test set with 3000 image
pairs. Note that this dataset does not contain any image of two different
models wearing same garment. Therefore, our test set does not have any ground
truth images. The training set contains 20,034 same person image pairs. We
chose MPV over the other benchamrk datasets DeepFashion [42] and VITON [7],
because DeepFashion [42] does not contain the separate garment images. Also
MPV and VITON both are collected from same source website so have similar data
and even overlap in contents.

###  IV-A Quantitative analysis

We quantitatively evaluate our results on two metrics - Fréchet Inception
Distance (FID) [43], the structural similarity metric (SSIM). FID measures the
similarity between two sets of images based on the features extracted from a
layer of inception v3 model [44] pretrained on ImageNet [45]. A lower value of
FID indicates better results. SSIM [46] is an image similarity metric, ranges
in [0, 1] (1 indicates same images).

We compare our performance with, CP-VTON [6], MGVTON [11], LGVTON [8] 777Note
that the results of LGVTON are reported based on human landmark annotations
only because fashion landmarks are not available in the MPV dataset., Roy et
al. [9], ACGPN [15] and He et al. [20] and report the SSIM and FID scores in
Table. I (SSIM score is reported only for the datasets containing ground
truth). Comparison with some other benchmarks PFAFN [29] and PASTA-GAN [33]
are presented in Table. II. Note that due to difficulty in execution of PASTA-
GAN’s official implementation we provide our method’s FID score on PASTA-GAN’s
MPV test set (provided by the authors). Note that in order to make fare
comparison we compare with almost equal number of M2P and C2P methods. M2P
methods - LGVTON, Roy et al., PASTA-GAN, PFAFN and C2P methods - MGVTON, CP-
VTON, ACGPN, He et al. Also like ours, both LGVTON and PASTA-GAN uses model
and person’s pose key points in computing the target warp.

The C2P methods [11, 6, 15, 20] take separate garment image as input. Such
garments are kept in one pose only and also have no occluded parts. But in
case of model image the clothing might be occluded and also pose variability
between the source and the target person might be there. To make fair
comparison we execute these methods on both clean garment image and segmented
garment image. Results in Table. I shows that our method outperforms the most
of the other methods in terms of both metrics in both the input setting.
Though He et al. achieves better score in case of no occlusion and no source
clothing’s pose variability. It is also observable in Table. II that also on
the MPV test set of PASTA-GAN our method scores better compared to others.

TABLE I: Quantitative evaluation on different datasets. In column 2 the inputs of MGVTON, CP-VTON, ACGPN, and He et al. is ($c$, $P$) and that of the others is ($M$, $P$). In column 5, MGVTON, CP-VTON, ACGPN, and He et al. are run in their default inputs setting i.e., is ($C$, $P$). Since in the training set source and the target images are same, therefore no variation in pose, so, no question of occlusion, hence result will be same for both the clothing sources.  | Test Set | Training Set | Test Set
---|---|---|---
| (M2P) |  | (C2P)
Method | FID$\downarrow$ | FID$\downarrow$ | SSIM$\uparrow$ | FID$\downarrow$
MGVTON(ICCV, 19) | 47.23 | 35.70 | 0.76 | 38.19
CP-VTON (ECCV, 18) | 50.67 | 21.03 | 0.74 | 40.02
LGVTON(MTA, 22) | 28.86 | 12.06 | 0.89 | -
Roy et al.(IVCNZ, 2020) | 28.41 | 14.34 | 0.80 | -
ACGPN (CVPR, 20) | 27.80 | 14.35 | 0.88 | 17.87
He et al.(CVPR, 22) | 18.31 | 11.45 | 0.89 | 9.38
Ours | 16.38 | 9.45 | 0.93 | -
TABLE II: The FID scores on the MPV test set used in PASTA-GAN. The scores of all the methods except ours are taken from the paper PASTA-GAN.  | Test Set
---|---
| (M2P)
Method | FID$\downarrow$
CP-VTON (ECCV, 18) | 37.72
ACGPN (CVPR, 20) | 23.20
PFAFN (CVPR, 21) | 17.40
PASTA-GAN (NeurlIPS, 21) | 16.48
Ours | 15.74

Figure 7: Comparative study on the cases of significant arm bending.

Figure 8: Qualitative comparison of the proposed MPN with the semantic
generation module (SGM) of ACGPN. We have highlighted the areas to be noticed
in each of the results. Observe in most of the cases SGM does not preserve the
features of the source clothing in the try-on result, instead keeps those
features same as the old cloth of the person (i.e., before try-on).

###  IV-B Qualitative Analysis

Some qualitative comparisons are given in Fig. 7. Note that the limitations of
the methods when the segmented clothing image is given as input can be
verified from the quantitative analysis done before. Here we chose to compare
in every method’s default input setting i.e, the garment image as input for
[15, 11, 6, 20] and the model image to [8]. This is to compare all the methods
at the best of their ability.

It can be observed that while handling critical hand bending all the methods
except ours fails to perform well. In case of CP-VTON and MGVTON probably the
poor result occurs due to impaired learning of human body geometry by the
geometric matching network. ACGPN shows some improvement due to the second-
order difference constraint. Notice that CP-VTON and ACGPN have tried to
compensate for the limitations of their warping method in their try-on stage.
For instance, the blurry areas in their predicted warps which are mostly
because those regions are being filled by the try-on synthesis module of the
respective methods which are not strong enough to fill the texture or color
accurately. In comparison to others, our results show better-predicted warps.
Similar to LGVTON employing the idea of landmarks correspondence in warping
produces correct deformation in the torso region. However, unlike the others,
we can handle the bending of the sleeves due to our geometric warping
approach; and also our part-by-part warping approach enables us in handling
the cases of overlap between sleeve and torso. We also present a visual
comparison among the results of our MPN and the corresponding semantic
generation module (SGM) of ACGPN in Fig. 8 because the objectives of these
modules are same. The results show some significant features (e.g., neck
pattern, sleeve length, highlighted in Fig. 8) of the target clothing mask are
predicted better by our MPN compared to SGM.

###  IV-C Ablation Study

We analyze the significance of employing MPN and its human parsing branch in
our approach. In the former case, we compute the warping results without using
the mask predicted by MPN $\mathcal{S}$, i.e., we only compute $X^{\prime}$
corresponding to the $X\in C_{sleeve}^{\prime}$ instead of
$X\in\\{C_{sleeve}^{\prime}\cap\mathcal{S}\\}$.

For the latter case, we trained an instance of MPN without the parsing branch
and compute the final results using that. Compared to the w/o MPN instance,
our method secures better score, which can also be verified from the visual
comparison given in Fig. 9, e.g., see the previously occluded parts are being
inpainted in target in our results. Compared to the without parsing instance,
our results secure a better FID score as shown in Table. III. A visual
comparison of the results are portrayed in Fig. 9, shows improvement in some
of the predicted feature details e.g., near the collar and in the adjacent
region of upper and lower body clothes we see improved details.

TABLE III: On the significance of MPN and MPN without parsing branch in the synthesis stage of our method. Method |  FID$\downarrow$
---|---
Ours (target warping w/o mask from MPN) |  16.69
Ours (w/o parsing branch in MPN) |  16.66
Ours |  16.38

Figure 9: illustration of the significance of MPN and MPN without parsing
branch in the synthesis stage.

##  V Conclusion

This work proposes a novel virtual try-on approach. Our method compared to the
previous approaches shows improvement in terms of handling complex postures of
the input person images. In the upper body of human the torso undergoes
limited deformation but human arms can move in variety of ways. During try-on
this might cause the target warp to have significant deformation. Most of the
previous methods employed one transformation function to compute the target
warp. But different parts should be warped independently to model the cases of
overlap and also due to the different natures of movement of different parts
of the body. To address this, we follow a part-based approach. We propose a
geometric feature-based warping method to warp the sleeve that overcomes
several issues of previous methods. Our warping method is guided by pose key
points and follows constrains of human body movements. We also propose two
learning-based modules that aids in computing the warp more efficiently and
synthesizes a seamless output while taking care of the occluded source
clothing regions. While this method shows improvement in performance but the
the computation of landmarks and parsing adds overhead. Considering the
utility of the proposed geometric features in future we plan to work towards
exploring its possibility towards guiding appearance flow based methods.

## References

  * [1] G. Pons-Moll, S. Pujades, S. Hu, and M. J. Black, “Clothcap: Seamless 4d clothing capture and retargeting,” _ACM Transactions on Graphics (TOG)_ , vol. 36, no. 4, p. 73, 2017.
  * [2] M. Sekine, K. Sugita, F. Perbet, B. Stenger, and M. Nishiyama, “Virtual fitting by single-shot body shape estimation,” in _Int. Conf. on 3D Body Scanning Technologies_. Citeseer, 2014, pp. 406–413.
  * [3] A. Mir, T. Alldieck, and G. Pons-Moll, “Learning to transfer texture from clothing images to 3d humans,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7023–7034.
  * [4] B. L. Bhatnagar, G. Tiwari, C. Theobalt, and G. Pons-Moll, “Multi-garment net: Learning to dress 3d people from images,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 5420–5430.
  * [5] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “Smpl: A skinned multi-person linear model,” _ACM transactions on graphics (TOG)_ , vol. 34, no. 6, pp. 1–16, 2015.
  * [6] B. Wang, H. Zheng, X. Liang, Y. Chen, L. Lin, and M. Yang, “Toward characteristic-preserving image-based virtual try-on network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 589–604.
  * [7] X. Han, Z. Wu, Z. Wu, R. Yu, and L. S. Davis, “Viton: An image-based virtual try-on network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7543–7552.
  * [8] D. Roy, S. Santra, and B. Chanda, “Lgvton: a landmark guided approach for model to person virtual try-on,” _Multimedia Tools and Applications_ , pp. 1–37, 2022.
  * [9] ——, “Incorporating human body shape guidance for cloth warping in model to person virtual try-on problems,” _2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ)_ , pp. 1–6, 2020.
  * [10] R. Yu, X. Wang, and X. Xie, “Vtnfp: An image-based virtual try-on network with body and clothing feature preservation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 10 511–10 520.
  * [11] H. Dong, X. Liang, X. Shen, B. Wang, H. Lai, J. Zhu, Z. Hu, and J. Yin, “Towards multi-pose guided virtual try-on network,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9026–9035.
  * [12] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [13] C.-W. Hsieh, C.-Y. Chen, C.-L. Chou, H.-H. Shuai, and W.-H. Cheng, “Fit-me: Image-based virtual try-on with arbitrary poses,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 4694–4698.
  * [14] C.-W. Hsieh, C.-Y. Chen, C.-L. Chou, H.-H. Shuai, J. Liu, and W.-H. Cheng, “Fashionon: Semantic-guided image-based virtual try-on with detailed human and clothing information,” in _Proceedings of the 27th ACM International Conference on Multimedia_. ACM, 2019, pp. 275–283.
  * [15] H. Yang, R. Zhang, X. Guo, W. Liu, W. Zuo, and P. Luo, “Towards photo-realistic virtual try-on by adaptively generating-preserving image content,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [16] A. H. Raffiee and M. Sollami, “Garmentgan: Photo-realistic adversarial fashion transfer,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 3923–3930.
  * [17] S. Jandial, A. Chopra, K. Ayush, M. Hemani, B. Krishnamurthy, and A. Halwai, “Sievenet: A unified framework for robust image-based virtual try-on,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 2182–2190.
  * [18] J. A. Maintz and M. A. Viergever, “A survey of medical image registration,” _Medical image analysis_ , vol. 2, no. 1, pp. 1–36, 1998.
  * [19] F. L. Bookstein, “Principal warps: Thin-plate splines and the decomposition of deformations,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 11, no. 6, pp. 567–585, 1989.
  * [20] S. He, Y.-Z. Song, and T. Xiang, “Style-based global appearance flow for virtual try-on,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3470–3479.
  * [21] Z. Wu, G. Lin, Q. Tao, and J. Cai, “M2e-try on net: Fashion from model to everyone,” in _Proceedings of the 27th ACM International Conference on Multimedia_. ACM, 2019, pp. 293–301.
  * [22] T. Liu, J. Zhang, X. Nie, Y. Wei, S. Wei, Y. Zhao, and J. Feng, “Spatial-aware texture transformer for high-fidelity garment transfer,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 7499–7510, 2021.
  * [23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [24] N. Jetchev and U. Bergmann, “The conditional analogy gan: Swapping fashion articles on people images,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2287–2292.
  * [25] G. Yildirim, N. Jetchev, R. Vollgraf, and U. Bergmann, “Generating high-resolution fashion model images wearing custom outfits,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [26] Y. Men, Y. Mao, Y. Jiang, W.-Y. Ma, and Z. Lian, “Controllable person image synthesis with attribute-decomposed gan,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5084–5093.
  * [27] K. M. Lewis, S. Varadharajan, and I. Kemelmacher-Shlizerman, “Tryongan: body-aware try-on via layered interpolation,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 4, pp. 1–10, 2021.
  * [28] A. Chopra, R. Jain, M. Hemani, and B. Krishnamurthy, “Zflow: Gated appearance flow-based virtual try-on with 3d priors,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5433–5442.
  * [29] Y. Ge, Y. Song, R. Zhang, C. Ge, W. Liu, and P. Luo, “Parser-free virtual try-on via distilling appearance flows,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8485–8493.
  * [30] X. Han, X. Hu, W. Huang, and M. R. Scott, “Clothflow: A flow-based model for clothed person generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 10 471–10 480.
  * [31] T. Beier and S. Neely, “Feature-based image metamorphosis,” _ACM SIGGRAPH computer graphics_ , vol. 26, no. 2, pp. 35–42, 1992.
  * [32] R. Alp Güler, N. Neverova, and I. Kokkinos, “Densepose: Dense human pose estimation in the wild,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7297–7306.
  * [33] Z. Xie, Z. Huang, F. Zhao, H. Dong, M. Kampffmeyer, and X. Liang, “Towards scalable unpaired virtual try-on via patch-routed spatially-adaptive gan,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 2598–2610, 2021.
  * [34] W. Liu, Z. Piao, J. Min, W. Luo, L. Ma, and S. Gao, “Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5904–5913.
  * [35] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4401–4410.
  * [36] K. Gong, X. Liang, D. Zhang, X. Shen, and L. Lin, “Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 932–940.
  * [37] K. Lin, L. Wang, K. Luo, Y. Chen, Z. Liu, and M.-T. Sun, “Cross-domain complementary learning using pose for multi-person part segmentation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2020.
  * [38] F. Malagelada, M. Dalmau-Pastor, J. Vega, and P. Golano, “Elbow anatomy,” _Sports injuries: prevention, diagnosis, treatment and rehabilitation_ , vol. 2, pp. 527–53, 2014.
  * [39] V. Masteikaitě, V. Sacevičienė, and V. Čironienė, “Compressed loop method for the bending behaviour of coated and laminated fabrics analysis,” _Journal of Industrial Textiles_ , vol. 43, no. 3, pp. 350–365, 2014.
  * [40] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime multi-person 2d pose estimation using part affinity fields,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [41] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 85–100.
  * [42] Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang, “Deepfashion: Powering robust clothes recognition and retrieval with rich annotations,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1096–1104.
  * [43] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6626–6637.
  * [44] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [45] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in _CVPR09_ , 2009.
  * [46] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli _et al._ , “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004\.


# Sequential Graph Attention Learning for Predicting Dynamic Stock Trends
(Student Abstract)

Tzu-Ya Lai1, Wen Jung Cheng2, Jun-En Ding3 Corresponding Author

###### Abstract

The stock market is characterized by a complex relationship between companies
and the market. This study combines a sequential graph structure with
attention mechanisms to learn global and local information within temporal
time. Specifically, our proposed “GAT-AGNN” module compares model performance
across multiple industries as well as within single industries. The results
show that the proposed framework outperforms the state-of-the-art methods in
predicting stock trends across multiple industries on Taiwan Stock datasets.

## Introduction

The stock market can be considered a dynamic system with numerous intricately
connected parts that change over time. Graph Attention Networks (GATs) and
Gated Recurrent Units (GRUs) have been used in some research to capture
temporal information of the stock exchange due to the rapid growth of Graph
Neural Networks (GNNs) (hsu2021fingat, hsu2021fingat). Each company is
considered a node in the graph in these studies, and the correlation between
two nodes at a certain time point determines the edge between them. In order
to aggregate data on momentum spillover between companies, cheng2021modeling
(cheng2021modeling) previously proposed a model that integrates Graph
Attention Network (GAT) and Attribute-Mattered Aggregator. In this paper, we
argue that: a) the majority of research does not confirm the efficacy of the
graph neural network approach, including whether it can be data-driven to
capture company interrelationships; and b) the single-layer feed-forward
neural network used as the Attribute-Mattered Aggregator in previous research
(cheng2021modeling, cheng2021modeling) should be converted to a more
methodologically sophisticated neural network module. We propose a novel model
framework called the GAT-AGNN model to enhance prediction performance. We also
suggest implementing the Corr-Cos model to confirm the efficacy of the graph
neural network approach.

#### Problem Definition

This study aims to forecast return on stocks across multiple industries based
on their histories. We focus on two tasks founded in our analysis of return
ratios from daily trading information: a) classification problems (classifying
the future stock return from the closing price as a positive return or
negative return); and b) regression problems (predicting future stock closing
price return).

#### Data Collection

Our research uses daily data from _Taiwan Economic Journal_ (TEJ) to predict
the stock market trend. Every company has a highest/lowest price,
opening/closing price and turnover rate. Data was collected from November 18,
2019, to June 20, 2022; a total of 630 days. The 30 companies studied were
selected from the Yuanta/P-shares Taiwan Mid-Cap 100 ETF.

## Proposed Method

### Sequential Fusion Module with GAT and AGNN

#### Gated Recurrent Unit (GRU)

Given the $i-th$ company, and the independent variable for time series
representation learning is
$\mathcal{X}_{i}^{[t-T:t)}=[\begin{bmatrix}\mathbf{x}_{i}^{t-T},&\cdots,&\mathbf{x}_{i}^{t-1}\end{bmatrix}]^{\mathsf{T}}$
which is the historical data for the first $T$ days of the $i-th$ firm, where
$T$ is the length of the rolling window. This research adopts GRU as the
module for time series data extraction.

$\mathbf{h}_{i}^{t}=GRU(\mathcal{X}_{i}^{[t-T:t)}).$ (1)

The historical feature vector $\mathbf{h}_{i}^{t}$ will be used to retrieve
data for the previous $T$ days in accordance with the approach described
above.

Our approach utilizes the GRU module to create a graph-structured dynamic
system (see Figure 1) that captures time-series dependency information for
each company, and the nodes that constitute this dynamic system are derived
from the fusion historical information vectors
$\mathcal{H}=[\begin{bmatrix}{\mathbf{h}_{1}^{t}},&\cdots,&{\mathbf{h}_{N}^{t}}\end{bmatrix}]$.

#### Graph Attention Network (GAT)

The prospective link between companies i and j at time t is therefore captured
by ${U_{i,j}}^{t}$.

$\mathcal{U}_{i,j}^{t}=u(\mathbf{h}_{i}^{t},\mathbf{h}_{j}^{t})=ELU({\mathbf{a}_{u}}^{\mathsf{T}}\mathcal{W}_{u}[{\mathbf{h}_{i}^{t}}^{\mathsf{T}}\mathbin{\|}{\mathbf{h}_{j}^{t}}^{\mathsf{T}}]),$
(2)

To identify prospective company relationships via the shared attention
mechanism, a single-layer neural network and an ELU activation function are
used to train the correlation between nodes $i$ and $j$. The softmax function
is applied to normalize all pairwise connections, and the correlation between
companies $(i,j)$, is normalized at time $t$, can be identified as follows:

$\widetilde{Q_{i,j}}^{t}=softmax_{j}=\frac{exp(\mathcal{U}_{i,j}^{t})}{\sum_{k{\in}N,k{\neq}i}exp(\mathcal{U}_{i,k}^{t})}.$
(3)

Figure 1: Diagram of GAT-AGNN’s architecture.

#### Attention-Based Graph Neural Network (AGNN)

Using calculation ${C_{i,j}}^{t}$ can capture potentially connected
organizations based on similarities between companies. By using a trainable
$\alpha$, the prospective connections between companies $i$ and $j$ at time
$t$, can be identified as follows:

$\mathcal{C}_{i,j}^{t}=c(\mathbf{h}_{i}^{t},\mathbf{h}_{j}^{t})=\alpha\cos(\mathcal{W}_{c}[{\mathbf{h}_{i}^{t}}^{\mathsf{T}},{\mathbf{h}_{j}^{t}}^{\mathsf{T}}]),$
(4)

The softmax function is used to normalize all pairwise connections.

$\widetilde{G_{i,j}}^{t}=softmax_{j}=\frac{exp(\mathcal{C}_{i,j}^{t})}{\sum_{k{\in}N,k{\neq}i}exp(\mathcal{C}_{i,k}^{t})}.$
(5)

#### Sequential Relation Module

The relational of the $i-th$ company at time $t$ information from the GRU
module vector is defined as
$\mathcal{V}=[\begin{bmatrix}{\mathbf{v}_{1}^{t}},&\cdots,&{\mathbf{v}_{N}^{t}}\end{bmatrix}]$,
while ${\mathbf{v}_{i}^{t}}$ is the correlation sequential vector calculated
as follows:

$\mathbf{v}_{i}^{t}={\boldsymbol{tanh}(\sum_{j,j{\neq}i}^{N}\widetilde{Q_{i,j}}^{t}W_{s}{\mathbf{h}_{j}^{t}}^{\mathsf{T}}\otimes\widetilde{G_{i,j}}^{t}W_{v}{\mathbf{h}_{j}^{t}}^{\mathsf{T}})},$
(6)

In addition to the attention mechanism, we consider a multi-head option.

$\mathbf{v}_{i}^{t}=\mathop{\Big{\|}}\limits_{m=1}^{M}{\boldsymbol{tanh}(\sum_{j,j{\neq}i}^{N}{\widetilde{Q_{i,j}}}^{t,[m]}{W_{s}}^{[m]}{\mathbf{h}_{j}^{t}}^{\mathsf{T}}\otimes{\widetilde{G_{i,j}}}^{t,[m]}{W_{v}}^{[m]}{\mathbf{h}_{j}^{t}}^{\mathsf{T}})},$
(7)

The purpose of our prediction tasks are both Regression and Classification
tasks. First, a single-layer neural network with the function
$RELU({\mathcal{W}_{i}}^{\mathsf{T}}[{\mathbf{h}_{i}^{t}}^{\mathsf{T}}\mathbin{\|}{\mathbf{v}_{i}^{t}}]+\mathbf{b}_{i})$
is used; then, the second uses a single-layer neural network with the function
$Softmax({\mathcal{W}_{i}}^{\mathsf{T}}[{\mathbf{h}_{i}^{t}}^{\mathsf{T}}\mathbin{\|}{\mathbf{v}_{i}^{t}}]+\mathbf{b}_{i})$.

Additionally, we replace the Sequential Relation Module of GAT and AGNN with
the without training parameters module of Correlation coefficient and Cosine
similarity (Corr-Cos model). With these settings, we can test which modules
are more robust in their ability to learn complex information from industry
relationships.

$\mathbf{v}_{i}^{t}={\boldsymbol{tanh}(\sum_{j,j{\neq}i}^{N}\widetilde{Corr_{i,j}}^{t}W_{s}{\mathbf{h}_{j}^{t}}^{\mathsf{T}}\otimes\widetilde{Cos_{i,j}}^{t}W_{v}{\mathbf{h}_{j}^{t}}^{\mathsf{T}})}.$

Problems | Classification |  | Regression |
---|---|---|---|---
Metrics | ACC | AUC | MSE | MAE
LSTM | 0.5222 | 0.5154 | 1.7299 | 0.9591
LSTM+GAT | 0.5283 | 0.5023 | 1.3874 | 0.8633
GRU | 0.5139 | 0.5027 | 1.4963 | 0.8944
GRU+GAT | 0.5150 | 0.4805 | 1.2083 | 0.8031
Corr-Cos | 0.5328 | 0.5290 | 1.1850 | 0.7940
AD-GAT | 0.5189 | 0.5056 | 1.2143 | 0.8183
GAT-AGNN | 0.5350 | 0.5190 | 1.1585 | 0.7823
Table 1: Comparison of classification and regression prediction results across
models.

## Results and Discussion

In our research, we utilize different creating relationship functions (2) and
(4) to enhance and combine the attention mechanisms of GAT and AGNN for the
multi-head setting, which is superior to the traditional approach that relies
only on an activate function with a single-layer neural network.

| ACC | MSE
---|---|---
All Industry Industry6 1 | 0.5700 | 0.9851
Industry61 | 0.4767 | 1.2542

  * 1

Industry 6: shipping industry.

Table 2: Comparison of model performance across multiple industries and within
single industries. Figure 2: Stock time series forecasting visualization.
Time-dependent pattern captured by GAT-AGNN.

The results on the classification and regression prediction tasks are
displayed in Table1. According to the results below, the GAT-AGNN model
outperforms the Corr-Cos model, indicating that graph neural networks are
effective for such tasks. Moreover, the GAT-AGNN model lowers MSE by $5.5\%$
over the AD-GAT model (cheng2021modeling, cheng2021modeling). Table1 shows,
the GAT-AGNN model is not only significantly superior to GRU and GRU+GAT but
also LSTM and LSTM+GAT.

It is evident in Table 2 that the MSE in the multiple-industry model decreases
by 26.91%, while the ACC increases by 9.33% compared with the single-industry
model. Thus, our study suggests that sequential graph structure in
hierarchical information (see Figure 2) offers promise for the identification
of industry relations and information.

## References

* cheng2021modeling.
hsu2021fingat.

aaai22