# Taiji Data Challenge for Exploring Gravitational Wave Universe

Zhixiang Ren1,2, Tianyu Zhao1,3,211footnotemark: 1, Zhoujian
Cao1,3,422footnotemark: 2, Zong-Kuan Guo1,5,6,4,
Wen-Biao Han1,4,7,8,9,10, Hong-Bo Jin1,11,8, Yue-Liang Wu1,5,4,9

1 Taiji Laboratory for Gravitational Wave Universe,
University of Chinese Academy of Science (UCAS), Beijing 100049, China
2 PengCheng Laboratory, Shenzhen 518055, China
3 Department of Astronomy, Beijing Normal University, Beijing 100875, China
4 School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute
for Advanced Study,
UCAS, Hangzhou 310024, China
5 CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics,
Chinese Academy of Sciences, Beijing 100190, China
6 School of Physical Sciences, UCAS, Beijing 100049, China
7 Shanghai Astronomical Observatory, Chinese Academy of Sciences, Shanghai
200030, China
8 School of Astronomy and Space Science, UCAS, Beijing 100049, China
9 International Centre for Theoretical Physics Asia-Pacific (ICTP-AP, UNESCO),
UCAS, Beijing 100190, China
10 Shanghai Frontiers Science Center for Gravitational Wave Detection,
Shanghai 200240, China
11 Key Laboratory of Computational Astrophysics,
National Astronomical Observatories, Beijing 100101, China contributed
equallyCorresponding author(s<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The direct observation of gravitational waves (GWs) opens a new window for
exploring new physics from quanta to cosmos and provides a new tool for
probing the evolution of universe. GWs detection in space covers a broad
spectrum ranging over more than four orders of magnitude and enables us to
study rich physical and astronomical phenomena. Taiji is a proposed space-
based GW detection mission that will be launched in the 2030s. Taiji will be
exposed to numerous overlapping and persistent GW signals buried in the
foreground and background, posing various data analysis challenges. In order
to empower potential scientific discoveries, the Mock Laser Interferometer
Space Antenna (LISA) Data Challenge and the LISA Data Challenge (LDC) were
developed. While LDC provides a baseline framework, the first LDC needs to be
updated with more realistic simulations and adjusted detector responses for
Taiji’s constellation. In this paper, we review the scientific objectives and
the roadmap for Taiji, as well as the technical difficulties in data analysis
and the data generation strategy, and present the associated data challenges.
In contrast to LDC, we utilize second-order Keplerian orbit and second-
generation time delay interferometry techniques. Additionally, we employ a new
model for the extreme-mass-ratio inspiral waveform and stochastic GW
background spectrum, which enables us to test general relativity and measure
the non-Gaussianity of curvature perturbations. Furthermore, we present a
comprehensive showcase of parameter estimation using a toy dataset. This
showcase not only demonstrates the scientific potential of the Taiji Data
Challenge but also serves to validate the effectiveness of the pipeline. As
the first data challenge for Taiji, we aim to build an open ground for data
analysis related to Taiji sources and sciences. More details can be found on
the official website http://taiji-tdc.ictp-ap.org.

_Keywords_ gravitational wave $\cdot$ universe evolution $\cdot$ Taiji $\cdot$
data challenge

##  1 Introduction

Ground-based GW detectors have made remarkable achievements since the first
detection of GW150914 [1], nearly 100 GW events have been observed so far [2].
The discovery of GWs not only provides a more fundamental test on Einstein’s
theory of general relativity (GR) but also leads us to study deeply on the
nature of gravity and spacetime. In particular, GWs detection in space enables
us to probe the formation and structure of black holes from the early Universe
through the peak of star formation era and to understand how does the
intermediate mass seed black hole formed in the early universe and how the
seed black hole grows into the large or extreme-large black hole [3]. It will
also be helpful to explore whether the dark matter could form into the black
hole or primordial black hole could be the candidate of dark matter.

Due to the limitation by seismic noise and arm length, ground-based detectors
are not sensitive at frequencies lower than 10 Hz [4]. By moving the detector
to space, we can eliminate the seismic and gravity-gradient noise and open up
the frequency band of the GW within $[10^{-4},0.1]\mathrm{Hz}$. The first
planned space-based GW detection project is the laser interferometer space
antenna (LISA) [5], proposed by the European Space Agency (ESA) and the
National Aeronautics and Space Administration (NASA), which will be launched
in around 2037 111https://sci.esa.int/web/lisa. Furthermore, LISA has laid a
solid foundation for space-borne GW detection, including scientific
objectives, data analysis algorithms, and instrumental techniques.

During the early 2000s, Chinese scientists were also interested in space-borne
GW detection programs and considered developing independent missions. The
feasibility of space-based GW detection was studied by the Chinese Academy of
Sciences (CAS) in 2008, followed by the initial mission proposal presented in
2011 [6]. A half-million-kilometer arm length and a noise budget that is a
hundred times more sensitive than LISA at 0.1 Hz made this instrument
particularly sensitive to intermediate mass black hole mergers [7]. In 2012,
the first mission descoping attempt was reported to prefer a constellation of
three satellites separated each by about 3 million kilometers, coordinating
both technical and scientific goals, and the proposed three-step road map was
first presented in which the launch time for the Chinese space GW detector was
expected to be in the 2030s [8, 7]. The CAS built prototypes that were used to
conduct proof-of-principle experiments [9, 10, 11, 12]. Similar to LISA, Taiji
consists of three spacecraft (SC), and each SC follows a heliocentric orbit.
The three SC form an approximately 3 million-kilometer-long equilateral
triangle. There is a 20-degree trail between the center of mass of the
constellation and the Earth. The constellation is roughly $1\mathrm{AU}$
distant from the Sun. This caused a slight eccentricity in the orbits of the
three SC. With the stable constellation, Taiji’s sensitive frequency band is
$[10^{-4},0.1]\mathrm{Hz}$ [7]. A three-step road map of the Taiji program is:
1) launch a satellite Taiji-1 for individual technique validation,  2) launch
two satellites Taiji-2 for almost all techniques test, and  3) launch complete
Taiji three satellites [13, 14].  The Taiji program, with its three-step road
map, has received priority support from the CAS’s strategic priority research
program since 2016 for its pre-experimental study. Some key technologies have
been developed further in the studies [13, 15, 16, 17, 18, 19, 20]. Currently,
the first phase of the Taiji-1 on-orbit experiment has been successfully
completed [21, 22].

In this paper, we present the first Taiji Data Challenge (TDC). Specifically,
we review the scientific objectives and technical challenges during the data
processing procedure. Then, we visualize all TDC datasets and explain their
scientific potential and technical obstacles. We implement time-domain time
delay interferometry (TDI) 2.0 for detector response calculation, which makes
our data more realistic compared to LDC1’s original datasets. As the main
component of our data challenge, we generate datasets that contain only one
source and also datasets that contain multiple sources. We further generate an
extreme-mass-ratio inspiral (EMRI) waveform constructed under the general
parameterized metric, which allows us to test GR and investigate the non-
Gaussianity of curvature perturbations by observing the GW background it
induces.

The paper is organized as follows. In Section 2, we introduce the scientific
objectives and technical challenge of Taiji mission. In Section 3, we
demonstrate how to simulate Taiji instrumental noise, detector response, and
GW waveform respectively. In Section 4, we show the properties of each
dataset. The last section contains our discussions about future work and
features of our TDC website.

##  2 Scientific objective and technical challenge of Taiji mission

Taiji detectors will receive a large amount of GW signals from different
sources in the target frequency band of $[10^{-4},0.1]\mathrm{Hz}$ [13]. The
observatory will measure signals from a wide range of different sources
relevant to the astrophysical mechanism of the formation of black holes and
galaxies, the test of GR, and the study of cosmology, including massive black
holes binaries (MBHBs) at all redshifts; EMRIs; the inspiral of stellar-origin
binary black holes (SOBBHs); galactic binaries (GB); verification galactic
binaries (VGBs), and various stochastic GW background (SGWB) sources [7].

The mission’s primary objective is to determine how and when massive black
holes have formed and grown over cosmic time. For the purpose of
reconstructing their evolution, the study will explore almost all of the mass-
redshift parameter space that is relevant [21]. We will investigate the
distribution of MBHB’s spins and masses in order to distinguish between
different formation mechanisms based on the GW signal from MBHBs [23].
Additionally, the mission will analyze the signals from GBs and provide
information on stellar evolution. Tens of millions of binaries in our galaxy
produce stochastic foregrounds, which we can estimate by observing GBs [24].
It is possible to constrain the evolutionary pathways of compact binaries by
examining the characteristics of the population, such as the number density of
sources as a function of frequency. It is expected that Taiji will provide
exceptionally strong tests of GR predictions by observing highly relativistic
MBHBs and EMRIs [5]. During the merger phase of a binary black hole system,
BHs travel at nearly the speed of light and interact strongly with each other,
which allows the study of the full nonlinear dynamics of gravity [25]. By
observing the signal of the EMRI system, the mass, spin, and quadrupole moment
of the central massive black hole will be measured making testing its level of
Kerr-ness and proving the no-hair hypothesis possible [26]. Finally, a space-
based GW detector will explore new physics and cosmology and seek out
previously unknown GW sources. The scientific objectives of the Taiji mission
are listed in Table 1.

Table 1: Scientific objectives of Taiji mission No. |  Scientific objective
---|---
1 |  Dynamical evolution and population of MBHBs, study the birth and growth of MBHs, and the astrophysical environment of the host galaxy
2 |  Precisely estimate parameters of MBH, reveals the physical nature of BHs, probe dynamics of galactic nuclei, and the astrophysical environment at the galaxy center by EMRI
3 |  Formation evolution and population of Galaxy binaries
4 |  Study SOBHB formation, environment, population, and joint observation with LIGO
5 |  Test GR, study the properties of GW propagation
6 |  GW cosmology, i.e. measurement of the Hubble constant and cosmology constant
7 |  PSD shape and upper limit of SGWB signals
8 |  Detect unmodeled signals
9 |  New physics and cosmology beyond GR

GWs signals are expected to be many orders of magnitude below the noise level
caused by laser frequency fluctuations [5]. A post-processing technique called
TDI has been proposed to reduce laser noise to an acceptable level [27]. The
idea is to construct virtual equal-arm interference by combining the data
streams in the appropriate way. The second-generation TDI is applicable to
rotating and flexing detectors with linearly varying arm lengths. The actual
Taiji orbits are affected by the gravitational field of the planets, which
leads to non-periodic orbits. In science analysis, it is necessary to extract
scientific information from the various TDI data streams; in a similar manner
to the procedure for the electromagnetic case, science analysis is carried out
by deconvolution of the response function from the data stream in order to
obtain the incident GW field observed by Taiji.

The main task of the data analysis is to quickly identify the signals and
estimate their physical parameters precisely. Unlike current ground-based
detectors, Taiji will be sensitive to continuous sources, primarily from GBs.
It is expected that GBs will be the most numerous of the sources, the
unresolved component of the galactic population produces an effectively
stochastic "confusion noise". Due to the non-Gaussianity of GB foreground
noise, detecting other signals buried in the foreground noise is a challenging
task. What’s more, analysis of the foreground itself is another challenge
because of the enormous number of signals. Because of the long duration and
complex morphology of their signals, detecting and characterizing the physics
of SOBBHs with LISA is a highly nontrivial challenge. Data gaps will appear in
the Taiji data stream due to maintenance, telescope realignment, and
communication with Earth by a high-gain antenna. A series of significant
challenges arise when dealing with gapped data, namely the information loss,
spectral leakage, and noise correlations [28]. Another challenge for GW signal
detection is the interference of instrumental glitches, which can be separated
relatively natural from signals by ground observations with multiple
interferometers. In contrast, this coherent searching approach can not be
directly applied in space-based detection, so novel glitch classification
algorithms need to be developed [29]. All the technical challenges are
summarized in Table 2.

Table 2: Technical challenges of Taiji mission No. |  Technical challenges
---|---
1 |  Foreground GB signals separation
2 |  SOBBH signals separation
3 |  Others Signal with non-Gaussian GB noise
4 |  Overlapping MBHB signals
5 |  Instrumental glitches
6 |  Data gap due to maintenance, telescope re-alignment, and communication issue
7 |  TDI technique to suppress laser frequency noise

##  3 Method

When analyzing real detector data, it is necessary to model the signal and
noise; therefore, for simplicity in the TDC, several idealized models are
utilized during data simulation. This section will introduce these models and
their respective approximations. Using the existing waveform model, we first
obtain the source frame GW waveform during signal simulation. Then, we
determine the response of each link, which is dependent upon the orbit of the
Taiji SCs. In addition, TDI is used to obtain the signal in the X, Y, and Z
channels. The instrumental noise in those 3 channels is Gaussian noise
simulated by their PSD. In the final step, the signal is injected into the
noise.

###  3.1 Orbital motion

We use Keplerian orbit to approximate the motion of the Taiji SC [30, 31, 32].
Given the distance between the SC i.e. the arm length $L$ and the distance
from the SC to the sun $a$, the eccentricity of the orbit is defined by:

$\displaystyle
e=\left(1+\frac{4}{\sqrt{3}}\alpha\cos\phi+\frac{4}{3}\alpha^{2}\right)^{1/2}-1,$
(1)

where $\phi=\pi/3+5\alpha/8$ is the angle between the SC’s orbital planes and
the ecliptic plane. $\alpha=L/2a$ is a small parameter originally used in Ref.
[30]. The specific value of $\phi$ reduces the breathing effect of the arm
length [31].

The reference position of the 3 SC is given by:

$\displaystyle\left\\{\begin{aligned}
x_{\mathrm{ref},k}(t)&=a\cos\iota(\cos\psi_{k}(t)-e),\\\
y_{\mathrm{ref},k}(t)&=a\sqrt{1-e^{2}}\sin\psi_{k}(t),\\\
z_{\mathrm{ref},k}(t)&=-a\sin\iota(\cos\psi_{k}(t)-e).\end{aligned}\right.$
(2)

where $\iota$ is the orbital inclination of SC1 relative to the ecliptic
plane.

$\displaystyle\tan\iota=\frac{2}{\sqrt{3}}\frac{\alpha\sin\phi}{\left(1+\frac{2}{\sqrt{3}}\alpha\cos\phi\right)},$
(3)

where $\psi_{k}(t)$ is the eccentric anomaly of $\mathrm{SC}_{k}$, which is
obtained by solving the Kepler equation to the desired order iteratively:

$\psi_{k}(t)-e\sin\psi_{k}(t)=m_{k}(t)=m_{0,k}+\Omega(t-t_{0})=m_{0,1}-\frac{2\pi(k-1)}{3}+\Omega(t-t_{0}),\quad\Omega=\frac{2\pi}{1\
\mathrm{year}}.$ (4)

Then, the position of 3 SC could be expressed in terms of the reference
position and periastron $\lambda_{k}=\lambda_{1}+2\pi(k-1)/3$:

$\displaystyle\left\\{\begin{aligned}
x_{k}(t)&=\cos\lambda_{k}x_{\mathrm{ref},k}(t)-\sin\lambda_{k}y_{\mathrm{ref},k}(t),\\\
y_{k}(t)&=\sin\lambda_{k}x_{\mathrm{ref},k}(t)+\cos\lambda_{k}y_{\mathrm{ref},k}(t),\\\
z_{k}(t)&=z_{\mathrm{ref},k}(t).\end{aligned}\right.$ (5)

In summary, the orbits of 3 SCs are determined by 4 parameters: the semi-major
axis $a$, the arm length $L$, $\rm{SC}_{1}$’s initial mean anomaly $m_{0,1}$,
and $\rm{SC}_{1}$’s initial periastron $\lambda_{1}$.

###  3.2 Light travel time

Following Ref. [33], we could use the Taylor expansion of the position of the
emitter SC to calculate the light travel time analytically. So the light
travel time could be expressed as:

$\displaystyle\mathrm{LTT}=\mathrm{LTT}^{(0)}+\mathrm{LTT}^{(1)}+\mathrm{LTT}^{(2)}+\mathrm{LTT}^{(\mathrm{Sh})}$
(6)

with

$\displaystyle\mathrm{LTT}^{(0)}(t)$
$\displaystyle=\dfrac{1}{c}|\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t)|$
(7) $\displaystyle\mathrm{LTT}^{(1)}(t)$
$\displaystyle=\dfrac{\vec{v}_{\mathrm{recv}}(t)}{c^{2}}\cdot(\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t))$
(8) $\displaystyle\mathrm{LTT}^{(2)}(t)$
$\displaystyle=\dfrac{|\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t)|}{c^{3}}\bigg{[}\vec{v}_{\mathrm{recv}}^{2}(t)-\vec{a}_{\mathrm{recv}}(t)\cdot(\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t))$
(9)
$\displaystyle\qquad+\left(\frac{\vec{v}_{\mathrm{recv}}(t)\cdot(\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t))}{|\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t)|}\right)^{2}\bigg{]},$
(10)

where the subscripts ’send’ and ’recv’ represent the emitter SC and the
receiver SC respectively. Finally, we consider the Shapiro delay caused by the
Sun’s gravitational field, which is a second-order effect [34]:

$\mathrm{LTT}^{(\mathrm{Sh)}}=\frac{R_{S}}{c}\cdot\ln{\frac{|\vec{x}_{\mathrm{send}}(t)|+|\vec{x}_{\mathrm{recv}}(t)|+|\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t)|}{|\vec{x}_{\mathrm{send}}(t)|+|\vec{x}_{\mathrm{recv}}(t)|-|\vec{x}_{\mathrm{recv}}(t)-\vec{x}_{\mathrm{send}}(t)|}},$
(11)

where $R_{S}$ is the Schwarzschild radius of the Sun.

###  3.3 Detector response

For the data generation, we use a time domain detector response and the GPU
accelerated code provided by [35], which enables us to generate a large amount
of data directly in the time domain. First, we transform the GW waveform from
the source frame to the solar-system barycenter (SSB) frame by

$\left[\begin{array}[]{c}h_{+}^{\mathrm{SSB}}\\\
h_{\times}^{\mathrm{SSB}}\end{array}\right]=\left[\begin{array}[]{cc}\cos
2\psi&-\sin 2\psi\\\ \sin 2\psi&\cos
2\psi\end{array}\right]\left[\begin{array}[]{l}h_{+}^{\mathrm{src}}\\\
h_{\times}^{\mathrm{src}}\end{array}\right]\text{. }$ (12)

Then, the length variation of one arm caused by the GW signal is:

$\displaystyle
H(t)=h_{+}^{\mathrm{SSB}}(t)\xi_{+}(\hat{\theta},\hat{\phi},\vec{n}(t))+h_{\times}^{\mathrm{SSB}}(t)\xi_{\times}(\hat{\theta},\hat{\phi},\vec{n}(t)),$
(13)

where the antenna pattern $\xi_{+}$ and $\xi_{\times}$ are given by:

$\left\\{\begin{aligned}
\xi_{+}(\vec{u},\vec{v},\vec{n})&={\left(\vec{u}\cdot\vec{n}\right)}^{2}-{\left(\vec{v}\cdot\vec{n}\right)}^{2},\\\
\xi_{\times}(\vec{u},\vec{v},\vec{n})&=2\left(\vec{u}\cdot\vec{n}\right)\left(\vec{v}\cdot\vec{n}\right),\end{aligned}\right.$
(14)

$\hat{n}(t)$ is the unit vector of the arm, $\hat{\theta},\hat{\phi}$ are
polar and azimuthal angle in the SSB frame, based on the orthonormal basis
vectors $(\hat{e}_{r},\hat{e}_{\theta},\hat{e}_{\phi})$. $\vec{u},\vec{v}$ are
polarization vectors defined by $\vec{u}=-\hat{e}_{\phi}$ and
$\vec{v}=-\hat{e}_{\theta}$. The propagation vector is $\vec{k}=-\hat{e}_{r}$.
Light travel time along the arm affected by the GW is :

$\displaystyle t_{\mathrm{recv}}\simeq
t_{\mathrm{send}}+\mathrm{LTT}-\frac{1}{2c}\int_{0}^{L}H(\vec{x}(\lambda),t(\lambda))\,d\lambda,$
(15)

where we use first order approximation as
$t(\lambda)=t_{\mathrm{send}}+\lambda/c,\vec{x}(\lambda)=\vec{x}_{\mathrm{send}}(t_{\mathrm{send}})+\lambda\vec{n}(t_{\mathrm{send}})$,
$\vec{x}_{\mathrm{send}}$ is the position of the emitter SC. The parameter
$\lambda$ used as the integration variable describes the photon path. With
those approximations, $H(\vec{x}(\lambda),t(\lambda))$ is given by:

$H(\vec{x}(\lambda),t(\lambda))=H\left(t(\lambda)-\frac{\vec{k}\cdot\vec{x}(\lambda)}{c}\right)=H\left(t_{\mathrm{send}}-\frac{\vec{k}\cdot\vec{x}_{\mathrm{send}}(t_{\mathrm{send}})}{c}+\lambda\frac{1-\vec{k}\cdot\vec{n}(t_{\mathrm{send}})}{c}\right)$
(16)

Combining equation. (15) and (16) and differentiating the resulting expression
with respect to $t_{\mathrm{send}}$ yields the relative frequency shift,

$\displaystyle
y(t_{\mathrm{send}})=\frac{1}{2(1-\vec{k}\cdot\vec{n}(t_{\mathrm{send}}))}\left[H\left(t_{\mathrm{send}}-\frac{\vec{k}\cdot\vec{x}_{\mathrm{send}}(t_{\mathrm{send}})}{c}\right)-H\left(t_{\mathrm{send}}-\frac{\vec{k}\cdot\vec{x}_{\mathrm{recv}}(t_{\mathrm{recv}})}{c}+\mathrm{LTT}\right)\right].$
(17)

We assume that $\vec{n}(t_{\mathrm{send}})\simeq\vec{n}(t_{\mathrm{recv}})$
since the variation of arm direction due to the SC’s motion during the light
travel time is less than $10^{-3}\mathrm{rad}$ in the case of Taiji [36] and
$t_{\mathrm{recv}}\simeq t_{\mathrm{send}}+\mathrm{LTT}$. Finally, the GW
strain of one arm is:

$\displaystyle
y(t_{\mathrm{recv}})=\frac{1}{2(1-\vec{k}\cdot\vec{n}(t_{\mathrm{recv}}))}\left[H\left(t_{\mathrm{recv}}-\frac{\vec{k}\cdot\vec{x}_{\mathrm{send}}(t_{\mathrm{recv}})}{c}-\mathrm{LTT}(t_{\mathrm{recv}})\right)-H\left(t_{\mathrm{recv}}-\frac{\vec{k}\cdot\vec{x}_{\mathrm{recv}}(t_{\mathrm{recv}})}{c}\right)\right].$
(18)

###  3.4 TDI Combination

Due to the unequal arm length of the Taiji SCs, the laser frequency noise
could not be canceled naturally, which will be orders of magnitude above the
GW signals. So the TDI technique was developed to cancel the noise by shifting
the data from different SCs by subtle time delays and combining them together.
The TDI technique can be divided into several generations:

  * •

TDI 1.0 is used in the static case, which does not consider the rotation and
breathing effect. In other words, the arm lengths are constants in time, i.e.
the light travel time satisfies
$\mathrm{LTT}_{i}=\mathrm{LTT}_{i^{\prime}}=const$.

  * •

TDI 1.5 is used in the rigid rotating case, which does not consider the
breathing effect. The arm lengths are still constants in time, i.e.
$\mathrm{LTT}_{i}=const.,\mathrm{LTT}_{i^{\prime}}=const.$ but
$\mathrm{LTT}_{i}\neq\mathrm{LTT}_{i^{\prime}}$ here.

  * •

TDI 2.0 is used in the flexing case, the arm lengths are functions of time,
i.e.
$\mathrm{LTT}_{i}=\mathrm{LTT}_{i}(t),\mathrm{LTT}_{i^{\prime}}=\mathrm{LTT}_{i^{\prime}}(t)$,
and $\mathrm{LTT}_{i}(t)\neq\mathrm{LTT}_{i^{\prime}}(t)$.

Here we adopt the notation from Ref. [27], the number in the subscript
represents different links, where $i\in\\{1,2,3\\}$ denotes the clockwise link
opposite to $\rm{SC}_{i}$ and $i^{\prime}$ denotes the counter-clockwise link.
The first generation unequal arm Michelson combination is:

$\displaystyle
X_{1}(t)=\left(y_{2^{\prime}:322^{\prime}}+y_{1:22^{\prime}}+y_{3:2^{\prime}}+y_{1^{\prime}}\right)-\left(y_{3:2^{\prime}3^{\prime}3}+y_{1^{\prime}:3^{\prime}3}+y_{2^{\prime}:3}+y_{1}\right).$
(19)

The second-generation Michelson combination is

$\displaystyle
X_{2}(t)=y_{1^{\prime}}+y_{3,2^{\prime}}+y_{1,22^{\prime}}+y_{2^{\prime},322^{\prime}}+y_{1,3^{\prime}322^{\prime}}+y_{2^{\prime},33^{\prime}322^{\prime}}$
(20)
$\displaystyle\quad+y_{1^{\prime},3^{\prime}33^{\prime}322^{\prime}}+y_{3,2^{\prime}3^{\prime}33^{\prime}322^{\prime}}-y_{1}-y_{2^{\prime},3}-y_{1^{\prime},3^{\prime}3}-y_{3;2^{\prime}3^{\prime}3}$
$\displaystyle\quad-
y_{1^{\prime},22^{\prime}3^{\prime}3}-y_{3,2^{\prime}22^{\prime}3^{\prime}3}-y_{1;22^{\prime}22^{\prime}3^{\prime}3}-y_{2^{\prime},322^{\prime}22^{\prime}3^{\prime}3}.$

The variables $Y$ and $Z$ could be obtained by cyclic permutation of the
indices. The complete derivation and detailed explanation of these equations
can be found in Ref. [27]. The colon denotes time delay when the arm length is
time-independent,

$f(t)_{:j}:=f\left(t-\frac{L_{j}}{c}\right),\quad
f(t)_{:jk}=f\left(t-\frac{L_{k}}{c}-\frac{L_{j}}{c}\right)=f(t)_{:kj},$ (21)

which is commute for $j\neq k$. Next, a comma denotes time delays with time-
dependent arm lengths, here we compute chained delays as simple sums of delays
rather than nested delays,

$f(t)_{,jk}=f\left(t-\frac{L_{k}(t)}{c}-\frac{L_{j}(t)}{c}\right).$ (22)

This approximation is sufficient for computing the GW response function [35].

After performing the TDI combination, we obtain the X, Y, and Z channel
gravitational wave strain data. Simulating the instrumental noise is another
important aspect of the data simulation process, distinct from the strain
data. In the next subsection, we will delve into the detailed process of
simulating instrumental noise.

###  3.5 Instrumental noise

Despite the complex noise sources that will be involved in the Taiji detection
data in practice, in our simulation, we use a noise model consisting of just
two components for simplicity. The first one is the high-frequency component,
which represents the overall noises that come from the optical metrology
system, and the second is the low-frequency component, which represents the
test mass acceleration noise. The power spectral density (PSD) is given by:

$\displaystyle P_{\mathrm{oms}}(f)$ $\displaystyle=64\times
10^{-24}\frac{1}{\mathrm{Hz}}\left[1+\left(\frac{2\mathrm{mHz}}{f}\right)^{4}\right]\left(\frac{2\pi
f}{c}\right)^{2},$ (23) $\displaystyle P_{\mathrm{acc}}(f)$
$\displaystyle=9\times
10^{-30}\frac{1}{\mathrm{Hz}}\left[1+\left(\frac{0.4\mathrm{mHz}}{f}\right)^{2}\right]\left[1+\left(\frac{f}{8\mathrm{mHz}}\right)^{4}\right]\left(\frac{1}{2\pi
fc}\right)^{2}.$

Following [37], the analytic model of the one-sided noise PSD is derived from
the noise components in each interferometric measurement. We will explicitly
show PSD for a Michelson-type TDI generator. Given the data combination (19)
and (20) with the assumption that the noise of each link has the same PSD, the
noise PSD of the first and the second generation TDI X channel is:

$\mathrm{PSD}_{X_{1}}=16\sin^{2}(\omega
L)\left(P_{\mathrm{oms}}+(3+\cos(2\omega L))P_{\mathrm{acc}}\right),$ (24)
$\mathrm{PSD}_{X_{2}}=64\sin^{2}(\omega L)\sin^{2}(2\omega
L)\left(P_{\mathrm{oms}}+(3+\cos(2\omega L))P_{\mathrm{acc}}\right),$ (25)

where $\omega=2\pi f/c$. In this paper, we use TDI 2.0 PSD for noise
generation.

##  4 Dataset

###  4.1 Overall setting

The TDC dataset was designed based on the scientific objectives of Taiji. For
most of the sources of Taiji, we do not have well-established data analysis
algorithms because they are not detected by ground-based detectors. So at the
beginning of the data challenge, we should focus on each source separately for
simplicity. Each of the datasets from TDC-1 to TDC-6 contains signals from a
single source. Multiple sources are included in the last dataset TDC-7,
including MBHBs, VGBs, and GB foregrounds. Those datasets cover all the
sources for the scientific objectives. Similar to LDC, the sampling rate here
in TDC is 0.1 Hz for all datasets, while this setting will be adjustable with
Taiji’s configuration accordingly. First, we generate the signal, then compute
the TDI 2.0 response, and add it to the simulated noise. For the time domain
signals, we use the time domain TDI response as mentioned in 3.3 and 3.4. For
the stochastic GW signals, we generate them directly by their PSD, which
considers TDI response. Next, we will introduce the configuration and
parameters used to generate the datasets.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 1: Time-domain data of all the TDC datasets. The strain of TDI X
channel data and signals are presented. (a), TDC-1, the MBHB signal is zoomed
in because of its duration. (b) and (c), TDC-2-1 and TDC-2-2. (d), TDC-3. (e),
TDC-4. (f), TDC-5. (g) and (h), TDC-6-1 and TDC-6-2. (i), TDC-7, signals No.8
and No.9 are zoomed in due to the closeness of their coalescence time.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 2: Frequency domain data of all the TDC datasets. The PSD of TDI X
channel data and signals are compared with the noise PSD. (a), TDC-1. (b) and
(c), TDC-2-1 and TDC-2-2. (d), TDC-3. (e), TDC-4. (f), TDC-5. (g) and (h),
TDC-6-1 and TDC-6-2. (i), TDC-7.

###  4.2 Massive black hole binary

MBHBs are primary sources of Taiji, which is sensitive to MBHBs with total
mass in the range of $[10^{5},10^{8}]M_{\odot}$ [5]. It is expected to detect
$1\sim 20$ MBHBs per year [38]. We will be able to gain valuable insight into
how MBHs are formed by measuring their spins and masses. It should be possible
for us to test the hypothesis that modern galaxies are the result of the
merger of smaller "seed" galaxies in the early universe, as we will be able to
detect the merger of MBHB out to a high redshift. We use SEOBNRv4_opt [39]
here to generate the MBHB signal and shift it by the coalescence time $t_{c}$.
Only the $l=m=2$ mode is included in the data. Both TDC-1 and TDC-7 contain
MBHB signals. For TDC-1, there is only one MBHB signal in this dataset, the
time domain and the frequency domain behavior of the TDC-1 data are shown in
Fig. 1(a) and Fig. 2(a) respectively. In the time domain, we could see that
the merger portion of the signal is not buried in the noise, which has a short
duration of $\sim$ hour. Then, in the frequency domain, the signal is below
the noise PSD. The parameter of MBHB signals injected in those two datasets
are shown in Table 3, where $M_{T}$ is the total mass of two black holes, $q$
is the mass ratio with $q<1$, $s_{1},s_{2}$ is the spin parameters, $\iota$ is
the inclination angle, $\psi$ is the polarization angle, $\lambda$, and
$\beta$ is ecliptic longitude and ecliptic latitude characterize the sky
location of the source, $t_{c}$ is the coalescence time, and $\phi_{c}$ is
coalescence phase. TDC-1 only contains signal No. 5 and all 10 signals are
included in TDC-7. The time domain and the frequency domain behavior of the
TDC-7 data are shown in Fig. 1(i) and Fig. 2(i) respectively. Signals No. 8
and No. 9 are close together, presenting additional challenges to data
processing algorithms that are not encountered by ground-based detectors.

Table 3: Parameters of MBHB signals used in TDC-1 and TDC-7 No. | $M_{T}(M_{\odot})$ | $q$ | $s_{1}$ | $s_{2}$ |  $D_{L}$(Gpc) | $\iota$ | $\psi$ | $\lambda$ | $\beta$ | $t_{c}(yr)$ | $\phi_{c}$
---|---|---|---|---|---|---|---|---|---|---|---
1 | $1.2\times 10^{6}$ | 0.3 | 0.1 | 0.2 | 53.39 | $2\pi/3$ | $\pi/3$ | $\pi/10$ | $\pi/8$ | $0.2$ | 1
2 | $1.8\times 10^{6}$ | 0.8 | 0.14 | 0.12 | 16.02 | $\pi$ | $\pi/3$ | $\pi/15$ | $5\pi/4$ | $0.4$ | 0.05
3 | $2.4\times 10^{6}$ | 0.4 | 0.18 | 0.08 | 21.36 | $5\pi/6$ | $\pi/3$ | $\pi/20$ | $7\pi/4$ | $0.6$ | 0.1
4 | $1.5\times 10^{6}$ | 0.7 | 0.14 | 0.8 | 53.39 | $\pi/3$ | $\pi/3$ | $\pi/25$ | $3\pi/4$ | $0.8$ | 0.15
5 | $6\times 10^{5}$ | 0.5 | 0.2 | 0.4 | 53.39 | $\pi/3$ | $\pi/3$ | $\pi/5$ | $\pi/4$ | $1$ | 0.5
6 | $1.2\times 10^{6}$ | 0.6 | 0.1 | 0.6 | 106.78 | $\pi/6$ | $2\pi/3$ | $\pi/30$ | $5\pi/4$ | $1.2$ | 0.35
7 | $2.4\times 10^{6}$ | 0.1 | 0.22 | 0.48 | 16.02 | $4\pi/15$ | $5\pi/3$ | $\pi/35$ | $\pi/8$ | $1.4$ | 0.45
8 | $1.2\times 10^{6}$ | 0.05 | 0.16 | 0.28 | 26.69 | $13\pi/30$ | $\pi/3$ | $2\pi/5$ | $\pi/12$ | $1.6$ | 1
9 | $1.8\times 10^{6}$ | 0.5 | 0.02 | 0.08 | 32.03 | $14\pi/30$ | $\pi/3$ | $3\pi/5$ | $11\pi/40$ | $1.601$ | 0.4
10 | $1.2\times 10^{7}$ | 1 | 0.06 | 0.08 | 10.68 | $\pi/3$ | $\pi/3$ | $6\pi/5$ | $\pi/4$ | $1.8$ | 0.5

###  4.3 Extreme-mass-ratio inspiral

EMRIs are binary black hole systems consisting of a stellar-mass compact
object (CO) and a massive black hole with mass $M\sim
10^{4}M_{\odot}-10^{7}M_{\odot}$. Taiji will observe few tens to hundreds EMRI
events over 2 years with signal-to-noise ratio (SNR) above 20 [40]. Because of
the extremely large mass ratio, the EMRI evolves slowly, inspiraling about
$10^{4}\sim 10^{5}$ cycles in Taiji’s frequency band [41]. Since EMRI signals
contain rich information about the geometry surrounding the central black
holes, they are one of the primary fundamental physics goals of the Taiji
mission. It has been suggested that detections of EMRIs could provide a highly
accurate observational test of the "Kerr-ness" of the central MBH [42].What’s
more, EMRIs sample the stellar population near the central black holes.By
detecting EMRI signals, we can probe the physical properties of the central
MBH and test GR [5]. Several waveform templates are developed for such a
promising source of Taiji  1) Teukolsky-based method, [43, 44] 2) Numerical
Kludge (NK)[45],  3) Analytic Kludge (AK)[46],  4) Augmented Analytic Kludge
(AAK) [47, 48, 49], and  5) XSPEG [26, 50]. The first 2 of them are
computationally expensive, and the AK waveform doesn’t suit Kerr space-time,
so we utilize the GPU-accelerated AAK waveform [49] for the TDC-2-1 dataset
generation and the XSPEG waveform for the TDC-2-2 dataset generation. There is
only one EMRI signal in each dataset. Because of the KRZ metric used in XSPEG
waveform construction, TDC-2-2 could be used to test GR, which is a key
scientific objective of the Taiji mission. The time domain and the frequency
domain behavior of the data are shown in Fig. 1(b) and 2(c).Unlike MBHBs, the
amplitude of a typical EMRI is an order of magnitude below the instrumental
noise. Detection is only possible if enough SNR is accumulated over a number
of wave cycles. The parameters of EMRI waveform injected in TDC-2-1 is
$(M,\mu,a_{0},e_{0},p_{0},\iota_{0},\theta_{S},\phi_{S},\theta_{K},\phi_{K},\Phi_{\varphi,0},\Phi_{\theta,0},\Phi_{r,0},D_{L})=(10^{6}M_{\odot},30M_{\odot},0.6,0.6,15,0.7,10^{-6},10^{-6},10^{-6},10^{-6},0,0,0,100\mathrm{Mpc})$,
$M$ and $a$ are the mass and spin parameter of the MBH, $p$ is the semi-latus
rectum, $e$ is eccentricity, $\iota$ is the orbit’s inclination angle from the
equatorial plane, $\theta_{S}$, and $\phi_{S}$ are the polar and azimuthal sky
location angles. $\theta_{K}$ and $\phi_{K}$ are the azimuthal and polar
angles describing the orientation of the spin angular momentum vector of the
MBH. $\Phi_{\varphi,0},\Phi_{\theta,0},\Phi_{r,0}$, represent the phase of
azimuthal, polar, and radial modes respectively, $D_{L}$ is the luminosity
distance. The parameters of the EMRI waveform injected in TDC-2-2 is
$(M,\mu,a_{0},e_{0},p_{0},\iota_{0},\lambda,\beta,D_{L})=(10^{6}M_{\odot},10M_{\odot},0.9,0.6,15,\pi/4,0,\pi/4,8.3\mathrm{Mpc})$,
and the KRZ metric deformation parameter is
$(\delta_{1},\delta_{2},\delta_{3})=(0.1,0.1,0.1)$. Compared to the existing
LDC1-2 dataset for EMRI, an updated AAK waveform template is used in TDC-2-1,
and a waveform based on the KRZ metric is used in TDC-2-2.

###  4.4 Galactic binary

We use the simple GB model from Ref. [35], which is expressed in the source
frame as:

$\displaystyle h_{+}^{\mathrm{src}}(t)$
$\displaystyle=\mathcal{A}(1+\cos^{2}\iota)\cos\Phi(t),$ (26) $\displaystyle
h_{\times}^{\mathrm{src}}(t)$
$\displaystyle=2\mathcal{A}\sin\iota\sin\Phi(t),$ $\displaystyle\Phi(t)$
$\displaystyle=\phi_{0}+2\pi
f_{0}t+\pi\dot{f_{0}}t^{2}+\frac{\pi}{3}\ddot{f_{0}}t^{3},$
$\displaystyle\ddot{f_{0}}$
$\displaystyle=\frac{11}{3}\frac{\dot{f_{0}^{2}}}{f_{0}}.$

where $\mathcal{A}$ is the overall amplitude, $\phi_{0}$ is the initial phase
at the start of the observation, $\iota$ is the inclination of the BWD orbit
to the line of sight from the origin of the SSB frame. The intrinsic parameter
is the frequency of the signal $f$ and its derivative $\dot{f}$. The
comparison to LDC’s GB waveform $\ddot{f_{0}}$ is considered here.There are
enormous GBs populated in the Milky Way [5] which form foreground noise, and
disentangling those signals is a challenging task [24]. With the help of GWs,
GB foreground observations provide a key method for measuring the properties
of a fraction of the galactic stellar population that is routinely assumed but
has never been directly observed. What’s more, there are some bright GBs
called LISA verified binaries [51], the latest catalog is available
online222https://gitlab.in2p3.fr/LISA/lisa-verification-binaries. Following
the above catalog, we generate the TDC-3 dataset, which contains 43 VGBs. The
foreground GB noise is in the TDC-4 dataset, which consists of $\sim 3e7$ GB
signals. The catalog of the TDC-4 dataset comes from the population study of
GBs [52, 53].Both the signal and the noise PSD of these two datasets are shown
in Fig. 2(d) and 2(e), associated with the corresponding time domain data
shown in 1(d) and 1(e). Compared to LDC1’s dataset, we use TDI 2.0 here. That
GB noise is also included in TDC-7. The Taiji mission faces a data analysis
challenge because of the presence of non-Gaussian and non-stationary
astrophysical foregrounds. Due to the presence of foreground signals, GW
source separation procedures will be complicated. A resolvable signal’s source
parameter estimation will also be affected by the level of confusion noise.
Therefore, it is critical to study, understand, and predict the overall shape
and amplitude of the potential foreground components of the Taiji data.

###  4.5 Stellar-origin binary black holes

Space-based detectors observe the early inspiral of SOBBH systems, whose
merger will be detected by ground-based detectors. Those systems evolve slowly
in the LISA and Taiji frequency bands, which enable high-precision parameter
estimation [54]. The joint observation of SOBBH by Taiji and ground-based
detectors could be used to probe low-frequency modifications due to deviations
from GR or to environmental effects, to facilitate electromagnetic joint
observations [55, 25]. By estimating coalescence time and sky location, we
could forecast the signals for ground-based detectors. We generate the TDC-5
dataset, which contains 21721 SOBBH signals, whose catalog comes from [25].
The time domain and the frequency domain behavior of the data are shown in
Fig. 1(f) and 2(f). The SOBBH signals are completely below the noise in the
frequency domain, which are background signals. We generate the waveform using
the IMRPhenomD template, then transform it to the time domain and apply the
TDI response. The main difference between the TDC-5 and LDC1-5 datasets is
that we use the TDI 2.0 response in the time domain rather than the TDI 1.0
response in the frequency domain.

###  4.6 Stochastic gravitational wave background

One of the main scientific objectives of Taiji is to probe GWs from the early
universe and reveal physics of GW sources [56]. Fundamental physics and
astronomy would be greatly enhanced by the detection of a cosmological
background from the early universe. The only way to obtain direct information
about earlier epochs than decoupling may be through GWs and possibly
neutrinos. There are many possible astrophysical and cosmological sources that
contribute to the stochastic background, and up to now we put only upper
bounds on its amplitude [57], and on parameters characterizing its directional
properties [58], by the LIGO/Virgo collaboration. For space-based detection,
we just have a single detector, cross-correlation based method is not suitable
now, so a new SGWB detection algorithm is needed.

SGWBs that are Gaussian, isotropic, and stationary can be fully described by
their energy density spectrum [59]:

$\Omega_{GW}(f)=\frac{f}{\rho_{c}}\frac{{\rm d}\,\rho_{GW}}{{\rm d}\,f},$ (27)

where $\rho_{GW}$ is the energy density of gravitational radiation contained
in the frequency range $f$ to $f+df$, $\rho_{c}=\frac{3H_{0}^{2}c^{2}}{8\pi
G}$ is the critical density of the universe, where $c$ is the speed of light,
and $G$ is Newton’s constant, and $H_{0}$ is the Hubble constant. Typical SGWB
sources include compact binary coalescence[57], cosmic phase transitions [60],
reheating or preheating after inflation [61], and primordial scalar and tensor
perturbations from inflation [62]. There are 2 datasets that contain different
types of SGWB signals, TDC-6-1 is generated by the power-law spectrum:

$h^{2}\Omega_{GW}(f)=10^{\alpha}\left(\frac{f}{f_{*}}\right)^{\beta},$ (28)

where $h$ is the dimensionless Hubble constant, $f_{*}$ is the pivot
frequency, $\alpha$ characterize its amplitude at $f_{*}$ and $\beta$ is the
slope of the spectrum. The SGWB signal in TDC-6-1 is generated using the
parameters $\alpha=-12$ and $\beta=2/3$.

TDC-6-2 is generated by non-Gaussian scalar perturbation-induced SGWB signals.
The induced GWs are generated during the formation of primordial black holes
(PBHs), providing a powerful tool to search for PBHs [63]. Typically,
curvature perturbations are assumed to have Gaussian probability density
functions. There is a natural expectation that non-Gaussianity will be present
when a sharp peak appears in the power spectrum of curvature perturbations.
GWs induced by scalar perturbations with a second-order local-type non-
Gaussianity were studied in Ref. [62]. The formulae of the SGWB spectrum used
to generate TDC-6-2 data is given in equation (7) in Ref. [62] with the
parameters $F_{\mathrm{NL}}=10,\sigma=10^{-4},M_{\mathrm{PBH}}=10^{22}g$,
where $F_{\mathrm{NL}}$ characterize the non-Gaussianity, $\sigma$ is the peak
width of curvature perturbation spectrum, and $M_{\mathrm{PBH}}$ is the mass
of PBH with the assumption that PBHs can serve as all dark matter.

In summary, 9 datasets are designed for TDC, and their essential motivation
(i.e. the scientific objective and the technical challenge), and
configurations are presented in Table. 4. The scientific objectives and the
corresponding technical challenges are shown in Tab. 1 and Tab. 2.

Table 4: Summary of Taiji Data Challenge datasets Name | Scientific objective | Technical challenge | Signals | Waveform model
---|---|---|---|---
TDC-1 | 1, 5, 6 | 7 | 1 MBHB signal | SEOBNRv4_opt
TDC-2-1 | 2, 5, 6 | 7 | 1 EMRI signal | PN5AAK
TDC-2-2 | 2, 5, 6 | 7 | 1 EMRI signal | XSPEG
TDC-3 | 3, 5, 6 | 7 | 43 VGB signals | Sinusoidal
TDC-4 | 3, 5, 6 | 1, 7 |  $3\times 10^{7}$ GB signals | Sinusoidal
TDC-5 | 4, 5, 6 | 2, 7 |  $2\times 10^{5}$ SOBBH signals | IMRPhenomD
TDC-6-1 | 5, 6, 7 | 7 | SGWB | Power-law
TDC-6-2 | 5, 6, 7 | 7 | SGWB |  | Non-Gaussian
---
curvature
perturbations
TDC-7 | 1, 3, 5, 6 | 1, 3, 7 |  | 10 MBHBs, 43 VGBs,
---
$3\times 10^{7}\text{ GBs}$
| SEOBNRv4_opt,
---
Sinusoidal

##  5 Application

###  5.1 File format

The TDC consists of multiple datasets, with each dataset stored in a separate
HDF file., which can be accessed and downloaded from the dataset page on our
TDC website. Within the HDF file, all the data is organized under the
attribute TDIdata. Specifically, there is a data array of dimensions (4,
n_samples), where n_samples represents the total number of sampling points in
the time domain. Each row of the array corresponds to a specific aspect of the
data, namely time, TDI X strain, TDI Y strain, and TDI Z strain. This
structured arrangement ensures easy and efficient access to the desired
information within the dataset.

###  5.2 Toy dataset

Figure 3: MCMC posterior corner plot for the toy dataset. The MCMC posterior
analysis is conducted on our toy dataset, utilizing the following source
parameter values:
$(\mathcal{A},f,\dot{f},\lambda,\beta,\iota,\phi_{0},\psi)=(2.008192\times
10^{-23},0.00459585,1.469823\times
10^{-16},4.604082,-0.052144,0.39811,1.8777623071795864,2.927109)$. The
posterior distribution is represented by the dark cyan color, while the true
parameter values are indicated by the red lines.

In this subsection, we present a simple toy dataset, which provides
participants with an effective means to validate their pipelines and assess
the accuracy of their parameter estimation techniques. The toy dataset has
been intentionally designed to be straightforward, featuring only one
gravitational wave signal originating from a GB system. By simplifying the
dataset to a single signal, participants can precisely evaluate their
pipelines without the complexities introduced by waveform models or multiple
signals. This simplified setup provides a clear and unambiguous environment
for pipeline validation and establishes a solid foundation for accurate
parameter estimation. Using this toy dataset, participants have the
opportunity to rigorously evaluate the performance of their pipelines. They
can assess their pipelines’ ability to extract relevant parameters, such as
the gravitational wave frequency, amplitude, and arrival time. The well-
defined characteristics of the dataset allow participants to directly compare
their results, identify any discrepancies, and refine their pipelines
accordingly.

To facilitate the effective utilization of the toy dataset, we offer an
illustrative example of running a Markov Chain Monte Carlo (MCMC) parameter
estimation pipeline. The MCMC pipeline utilized in this demonstration is based
on the work by Falxa et al. [64], and the source code can be accessed at the
provided link 333 https://gitlab.in2p3.fr/lisa-apc/m3c2. This practical
demonstration serves as a comprehensive guide, showcasing how the dataset can
be leveraged to obtain reliable parameter estimates.

Furthermore, we showcase the outcomes of the parameter estimation process in
Fig. 3, illustrating the attained accuracy and precision through the
implementation of the MCMC pipeline on the toy dataset. The visual depiction
of the estimated parameters, juxtaposed with the corresponding true values,
offers a comprehensive overview of the pipeline’s performance and establishes
a benchmark for participants to assess and compare their own findings.

By introducing the practical demonstration of the MCMC pipeline and providing
visual representations of parameter estimation results, we not only equip
participants with the necessary tools to effectively utilize the toy dataset
but also prepare them for the real challenge posed by other TDC datasets. The
experience gained from working with the toy dataset serves as a valuable
cornerstone, allowing participants to build their skills and confidence before
tackling the complexities of the actual challenge datasets. The insights and
techniques acquired during this process will enable participants to face TDC
datasets with increased proficiency and contribute to advancements in space-
based gravitational wave data analysis.

##  6 Discussions

Building a data challenge is critical for scientists to develop algorithms and
pipelines for future data analysis tasks. We provide a complete workflow by
constructing detection data and waveform templates within and beyond GR. The
datasets and challenges are available on the official website of TDC
(http://taiji-tdc.ictp-ap.org). Scientists are encouraged to participate in
these data challenges and to submit their results. Currently, there is no
deadline, and users are free to submit their results at any time. We will rank
the submitted results for each challenge. As part of our website, we offer
both data downloads and data customization services, which enable users to
submit the data configuration file suited to their requirements, and we will
provide users links to download the customized dataset. With TDC, we hope to
create a community of researchers who can collaboratively contribute to the
development of Taiji’s data analysis pipelines and join the journey of
exploring the universe and making new discoveries with Taiji Collaboration.

##  7 Acknowledgements

The research was supported by the PengCheng Laboratory Cloud Brain and by
PengCheng Cloud-Brain. Further funding was provided by the National Key
Research and Development Program of China Grant No. 2021YFC2203001, No.
2020YFC2201501 & No. 2021YFC2203002, as well as the NSFC (No. 11920101003, No.
12021003, No. 12173071, No. 12147103, No. 12235019 and No. 12075297). Z.C and
W.H are supported by the CAS Project for Young Scientists in Basic Research
YSBR-006. Z.C was also supported by “The Interdisciplinary Research Funds of
Beijing Normal University".

## References

  * Abbott  _et al._ [2016] B. P. Abbott _et al._ , Phys. Rev. Lett. 116, 061102 (2016).
  * The LIGO Scientific Collaboration  _et al._ [2021] The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, _et al._ , “GWTC-3: Compact Binary Coalescences Observed by LIGO and Virgo During the Second Part of the Third Observing Run,” (2021), arXiv:2111.03606 .
  * González  _et al._ [2013] G. González, A. Viceré, and L. Wen, Front. Phys. 8, 771 (2013).
  * Matichard  _et al._ [2015] F. Matichard _et al._ , Class. Quantum Grav. 32, 185003 (2015).
  * Amaro-Seoane  _et al._ [2017] P. Amaro-Seoane _et al._ , “Laser Interferometer Space Antenna,” (2017), arXiv:1702.00786 .
  * Gong  _et al._ [2011] X. Gong, S. Xu, S. Bai, Z. Cao, _et al._ , Class. Quantum Grav. 28, 094012 (2011).
  * Luo  _et al._ [2021] Z. Luo, Y. Wang, Y. Wu, W. Hu, and G. Jin, Prog. Theor. Exp. Phys. 2021, 05A108 (2021).
  * Wu [2012] Y.-L. Wu, in  _Presentation to 1st eLISA Consortium Meeting_ (2012).
  * Li  _et al._ [2012] Y.-Q. Li, Z.-R. Luo, H.-S. Liu, Y.-H. Dong, and G. Jin, Chin. Phys. Lett. 29, 079501 (2012).
  * Liu  _et al._ [2014] H.-S. Liu, Y.-H. Dong, Y.-Q. Li, Z.-R. Luo, and G. Jin, Rev. Sci. Instrum. 85, 024503 (2014).
  * Dong  _et al._ [2014] Y.-H. Dong, H.-S. Liu, Z.-R. Luo, Y.-Q. Li, and G. Jin, Rev. Sci. Instrum. 85, 074501 (2014).
  * Li  _et al._ [2015] Y.-Q. Li, Z.-R. Luo, H.-S. Liu, Y.-H. Dong, and G. Jin, App. Phys. B 118, 309 (2015).
  * Hu and Wu [2017] W.-R. Hu and Y.-L. Wu, Natl. Sci. Rev. 4, 685 (2017).
  * Luo  _et al._ [2020] Z. Luo, Z. Guo, G. Jin, Y. Wu, and W. Hu, Results Phys. 16, 102918 (2020).
  * Luo  _et al._ [2017] Z. Luo, Q. Wang, C. Mahrdt, A. Goerth, and G. Heinzel, Appl. Opt. 56, 1495 (2017).
  * Luo  _et al._ [2018] Z. Luo, H. Liu, and G. Jin, Optics & Laser Tech. 105, 146 (2018).
  * Liu  _et al._ [2018a] H. Liu, Y. Dong, R. Gao, Z. Luo, and G. Jin, Optical Eng. 57, 1 (2018a).
  * Liu  _et al._ [2018b] H. Liu, Z. Luo, and G. Jin, Micrograv. Sci. and Technol. 30, 775 (2018b).
  * Deng  _et al._ [2018] W. Deng, T. Yang, J. Cao, E. Zang, L. Li, L. Chen, and Z. Fang, Optics Lett. 43, 1562 (2018).
  * Wang  _et al._ [2018] Z. Wang, W. Sha, Z. Chen, Y.-S. Kang, Z.-R. Luo, M. Li, and Y.-P. Li, Chinese Opt. 11, 131 (2018), (in Chinese).
  * The Taiji Scientific Collaboration [2021a] The Taiji Scientific Collaboration, Commun. Phys. 4, 34 (2021a).
  * The Taiji Scientific Collaboration [2021b] The Taiji Scientific Collaboration, Int J Mod Phys A 36, 2102002 (2021b).
  * Klein  _et al._ [2016] A. Klein, E. Barausse, A. Sesana, A. Petiteau, E. Berti, S. Babak, J. Gair, S. Aoudia, I. Hinder, F. Ohme, and B. Wardell, Phys. Rev. D 93, 024003 (2016).
  * Zhang  _et al._ [2021a] X.-H. Zhang, S. D. Mohanty, X.-B. Zou, and Y.-X. Liu, Phys. Rev. D 104, 024023 (2021a).
  * Sesana [2016] A. Sesana, Phys. Rev. Lett. 116, 231102 (2016).
  * Xin  _et al._ [2019] S. Xin, W.-B. Han, and S.-C. Yang, Phys. Rev. D 100, 084055 (2019).
  * Otto [2015] M. Otto,  _Time-Delay Interferometry Simulations for the Laser Interferometer Space Antenna_ , Ph.D. thesis, Hannover : Gottfried Wilhelm Leibniz Universität Hannover (2015).
  * Blelly  _et al._ [2021] A. Blelly, J. Bobin, and H. Moutarde, Mon. Notices Royal Astron. Soc. 509, 5902 (2021).
  * Robson and Cornish [2019] T. Robson and N. J. Cornish, Phys. Rev. D 99, 024019 (2019).
  * Dhurandhar  _et al._ [2005] S. V. Dhurandhar, K. R. Nayak, S. Koshti, and J.-Y. Vinet, Class. Quantum Grav. 22, 481 (2005).
  * Nayak  _et al._ [2006] K. R. Nayak, S. Koshti, S. V. Dhurandhar, and J.-Y. Vinet, Class. Quantum Grav. 23, 1763 (2006).
  * Wu  _et al._ [2019] B. Wu, C.-G. Huang, and C.-F. Qiao, Phys. Rev. D 100, 122001 (2019).
  * Chauvineau  _et al._ [2005] B. Chauvineau, S. Pireaux, T. Regimbau, and J.-Y. Vinet, Phys. Rev. D 72, 122003 (2005).
  * Hees  _et al._ [2014] A. Hees, S. Bertone, and C. Le Poncin-Lafitte, Phys. Rev. D 89, 064045 (2014).
  * Katz  _et al._ [2022] M. L. Katz, J.-B. Bayle, A. J. K. Chua, and M. Vallisneri, Phys. Rev. D 106, 103001 (2022).
  * Rijnveld and Pijnenburg [2017] N. Rijnveld and J. A. C. M. Pijnenburg, in  _International Conference on Space Optics — ICSO 2010_, edited by N. Kadowaki (SPIE, Rhodes Island, Greece, 2017) p. 96.
  * Babak  _et al._ [2021] S. Babak, M. Hewitson, and A. Petiteau, “LISA Sensitivity and SNR Calculations,” (2021), arXiv:2108.01167 .
  * Katz  _et al._ [2019] M. L. Katz, L. Z. Kelley, F. Dosopoulou, S. Berry, L. Blecha, and S. L. Larson, Mon. Notices Royal Astron. Soc. 491, 2301 (2019).
  * Bohé  _et al._ [2017] A. Bohé, L. Shao, _et al._ , Phys. Rev. D 95, 044028 (2017).
  * Gair  _et al._ [2017] J. R. Gair, S. Babak, A. Sesana, P. Amaro-Seoane, E. Barausse, C. P. L. Berry, E. Berti, and C. Sopuerta, J. Phys.: Conf. Ser. 840, 012021 (2017).
  * Babak  _et al._ [2017] S. Babak, J. Gair, A. Sesana, E. Barausse, C. F. Sopuerta, C. P. L. Berry, E. Berti, P. Amaro-Seoane, A. Petiteau, and A. Klein, Phys. Rev. D 95, 103012 (2017).
  * Barack and Cutler [2007] L. Barack and C. Cutler, Phys. Rev. D 75, 042003 (2007).
  * Glampedakis [2005] K. Glampedakis, Class. Quantum Grav. 22, S605 (2005).
  * Han and Cao [2011] W.-B. Han and Z. Cao, Phys. Rev. D 84, 044014 (2011).
  * Babak  _et al._ [2007] S. Babak, H. Fang, J. R. Gair, K. Glampedakis, and S. A. Hughes, Phys. Rev. D 75, 024005 (2007).
  * Barack and Cutler [2004] L. Barack and C. Cutler, Phys. Rev. D 69, 082005 (2004).
  * Chua and Gair [2015] A. J. K. Chua and J. R. Gair, Class. Quantum Grav. 32, 232002 (2015).
  * Chua  _et al._ [2017] A. J. K. Chua, C. J. Moore, and J. R. Gair, Phys. Rev. D 96, 044005 (2017).
  * Katz  _et al._ [2021] M. L. Katz, A. J. K. Chua, L. Speri, N. Warburton, and S. A. Hughes, Phys. Rev. D 104, 064047 (2021).
  * Zhang  _et al._ [2021b] C. Zhang, W.-B. Han, and S.-C. Yang, Commun. Theor. Phys. 73, 085401 (2021b).
  * Kupfer  _et al._ [2018] Kupfer _et al._ , Mon. Notices Royal Astron. Soc. 480, 302 (2018).
  * Nelemans  _et al._ [2001] G. Nelemans, L. R. Yungelson, and S. F. Portegies Zwart, Astron. Astrophys. 375, 890 (2001).
  * Nelemans and Tout [2005] G. Nelemans and C. A. Tout, Mon. Notices Royal Astron. Soc. 356, 753 (2005).
  * Gair  _et al._ [2021] J. Gair, M. Hewitson, A. Petiteau, and G. Mueller, in  _Handbook of Gravitational Wave Astronomy_, edited by C. Bambi, S. Katsanevas, and K. D. Kokkotas (Springer Singapore, Singapore, 2021) pp. 1–71.
  * Toubiana  _et al._ [2020] A. Toubiana, S. Marsat, S. Babak, J. Baker, and T. D. Canton, Phys. Rev. D 102, 124037 (2020).
  * Bartolo  _et al._ [2022] N. Bartolo, D. Bertacca, R. Caldwell, C. R. Contaldi, G. Cusin, V. De Luca, E. Dimastrogiovanni, M. Fasiello, D. G. Figueroa, G. Franciolini, A. C. Jenkins, M. Peloso, M. Pieroni, A. Renzini, A. Ricciardone, A. Riotto, M. Sakellariadou, L. Sorbo, G. Tasinato, J. Torrado, S. Clesse, and S. Kuroyanagi, J. Cosmol. Astropart. Phys. 2022, 009 (2022).
  * Abbott  _et al._ [2021] R. Abbott _et al._ (LIGO Scientific Collaboration, Virgo Collaboration, and KAGRA Collaboration), Phys. Rev. D 104, 022004 (2021).
  * The LIGO Scientific Collaboration  _et al._ [2022] The LIGO Scientific Collaboration, the Virgo Collaboration, and the KAGRA Collaboration, Phys. Rev. D 105, 122001 (2022).
  * Caprini  _et al._ [2019] C. Caprini, D. G. Figueroa, R. Flauger, G. Nardini, M. Peloso, M. Pieroni, A. Ricciardone, and G. Tasinato, J. Cosmol. Astropart. Phys. 2019, 017 (2019).
  * Hindmarsh  _et al._ [2014] M. Hindmarsh, S. J. Huber, K. Rummukainen, and D. J. Weir, Phys. Rev. Lett. 112, 041301 (2014).
  * Liu  _et al._ [2018c] J. Liu, Z.-K. Guo, R.-G. Cai, and G. Shiu, Phys. Rev. Lett. 120, 031301 (2018c).
  * Cai  _et al._ [2019] R.-G. Cai, S. Pi, and M. Sasaki, Phys. Rev. Lett. 122, 201101 (2019).
  * Yuan and Huang [2021] C. Yuan and Q.-G. Huang, Phys. Lett. B 821, 136606 (2021).
  * Falxa  _et al._ [2023] M. Falxa, S. Babak, and M. Le Jeune, Phys. Rev. D 107, 022008 (2023).


Minchul<EMAIL_ADDRESS>Sanghyuk<EMAIL_ADDRESS>Taeksoo<EMAIL_ADDRESS>Search Solutions Inc.
Republic of Korea Naver Corp.
Republic of Korea Naver Corp.
Republic of Korea Feature-Level Attribute Manipulation for Fashion

# Semi-supervised Feature-Level Attribute Manipulation for Fashion Image
Retrieval

###### Abstract

With a growing demand for the search by image, many works have studied the
task of fashion instance-level image retrieval (FIR). Furthermore, the recent
works introduce a concept of fashion attribute manipulation (FAM) which
manipulates a specific attribute (_e.g_ color) of a fashion item while
maintaining the rest of the attributes (_e.g_ shape, and pattern). In this
way, users can search not only “the same” items but also “similar” items with
the desired attributes. FAM is a challenging task in that the attributes are
hard to define, and the unique characteristics of a query are hard to be
preserved. Although both FIR and FAM are important in real-life applications,
most of the previous studies have focused on only one of these problem. In
this study, we aim to achieve competitive performance on both FIR and FAM. To
do so, we propose a novel method that converts a query into a representation
with the desired attributes. We introduce a new idea of attribute manipulation
at the feature level, by matching the distribution of manipulated features
with real features. In this fashion, the attribute manipulation can be done
independently from learning a representation from the image. By introducing
the feature-level attribute manipulation, the previous methods for FIR can
perform attribute manipulation without sacrificing their retrieval
performance.

##  1 Introduction

The fashion instance-level image retrieval (FIR) [Liu et al. (2012), Hadi
Kiapour et al. (2015), Gajic and Baldrich (2018), Kalantidis et al. (2013)] is
a task to find a clothing item that is exactly “the same” as a given query.
The FIR has been considered as a challenging task: the clothing items are
highly deformable, the viewpoint severely differs, and minor differences are
hard to catch. As an extension of FIR, the fashion attribute manipulation
(FAM) [Zhao et al. (2017), Ak et al. (2018a)] aims to find “similar” clothing
items by changing the unwanted attribute to the desired one, while preserving
the rest of the attributes. This feature is particularly important in fashion
search, because the users always desire to find clothing items with different
attributes such as shape, color, or pattern (i.e., the user might want to
search for a “white checked blouse,” not “blue”). To satisfy the user’s needs
in the fashion search, both FIR and FAM are important, and either one cannot
be sacrificed.

From this perspective, the methods so far can be divided into two classes: FIR
methods and FAM methods. The purpose of FIR methods is to find the same
instances as in a given query. Most FIR methods use the ranking loss based on
the instance labels for the training, sometimes exploiting the attribute
labels as well. Since these methods were inherently designed in consideration
of the FIR only, they are not capable of finding the item with desired
characteristics by manipulating the attributes of the fashion item [Ak et al.
(2018a)]. Meanwhile, FAM methods aim to find similar-looking items by
manipulating a specific attribute. Instead of creating pairs by instance
labels, FAM methods create pairs by attribute labels inducing a model to focus
on the attribute representation. However, creating pairs by attribute labels
requires a dense annotation [Zhao et al. (2017)], and having the same
combinations of attributes does not always guarantee that the positive
represents the same instance. For these reasons, FAM methods tend to sacrifice
a pure instance retrieval performance. The simplest way we can think for
solving both FIR, and FAM problem is to have both models for each specific
tasks. However, in real-life applications dealing with billion-scale images,
storing more than one features extracted from every single image is too
expensive. Instead, we designed a feature-level attribute manipulation method
that translates a learned feature directly into a manipulated feature
representing desired attributes.

Main contributions. In this paper, we propose a novel method that performs
search by feature-level attribute manipulation (FLAM). The contribution is
three fold: (1) We propose a new method for feature-level attribute
manipulation, and we demonstrate its effectiveness by achieving competitive
performance in both FIR and FAM tasks. (2) Unlike previous approaches, our
method no longer requires a dense annotation of attributes for training.
Instead, it transfers knowledge from learned attribute-specific embedders. (3)
We show that a model for attribute manipulation can be trained in an semi-
supervised manner, and a generative model can be used for image search with
attribute manipulation.

Figure 1: The specific architecture of FLAM, which consists of two
subnetworks: attribute-specific embedder, and query feature manipulator. The
figure shows the scenario of manipulating color attribute while preserving the
remaining attributes (shape and pattern). Note that the red-colored embedding
vector is extracted from the randomly-sampled real feature that owns the
desired color attribute (e.g., “feature of red color item, if the query is
blue ”).

##  2 Related Work

The previous research on image retrieval in fashion domain can be categorized
as follows: instance-based retrieval, attribute-based retrieval, and attribute
manipulation.

Instance-based retrieval. The recent instance-level image retrieval methods
have shown a dramatic increase in performance using advances in metric
learning [Gordo et al. (2017), Schroff et al. (2015), Sohn (2016), Yu et al.
(2018), Jun et al. (2019)]. The metric learning focuses on the way it
calculates loss by making a pair in an effective way. The instance ID is
required for making a pair, and no additional labels are needed in general. In
the fashion domain, several studies considered the cross-domain problem that
aims to find in-shop images by querying street images in uncontrolled
conditions [Liu et al. (2012), Hadi Kiapour et al. (2015), Gajic and Baldrich
(2018), Kalantidis et al. (2013)]. Liu et al. Liu et al. (2012) learns a
similarity measure between the street and in-shop domain, and Hadi Kiapour et
al. Hadi Kiapour et al. (2015) uses human/clothing part alignment. Gajic and
Baldrich Gajic and Baldrich (2018) trained a simple model with triplet loss
and max pooling.

Attribute-based retrieval. The image retrieval in fashion domain is a distinct
task in that fashion items have attributes. The attributes provide useful
information to understand the fashion image in detail Chen et al. (2012a);
Kovashka et al. (2012); Kiapour et al. (2014). For this reason, many publicly
available fashion datasets provide the attribute labels as well Liu et al.
(2016); Huang et al. (2015); Ak et al. (2018b); Corbiere et al. (2017). We
classify the methods that use attribute labels for training as attribute-based
retrieval. Liu et al. Liu et al. (2016) proposed FashionNet, which predicts
landmarks and attributes simultaneously. A triplet loss is used for metric
learning of pairwise clothes images. Huang et al. Huang et al. (2015) proposed
a dual attribute-aware ranking network, which uses attribute-guided learning
and triplet visual similarity constraint. Di et al. Di et al. (2013) defined a
set of style-related attributes and trained an individual classifier for each
style. The classifiers were then used for searching items by attributes.
Yamaguchi et al. Yamaguchi et al. (2015) proposed a joint clothing detection
model that combined a deep network with conditional random fields, considering
the compatibility of clothing items and attributes. Singh and Lee Singh and
Lee (2016) suggested an end-to-end training that localizes and ranks relative
visual attributes in a weakly supervised manner. While visual attributes of
fashion items provide high-level information, we found that the attribute
labels in most of the fashion datasets are severely noisy, which may interfere
with learning a good-quality image representation.

Attribute manipulation. Recently, a concept of interactive search in fashion
domain has been introduced Kovashka and Grauman (2013); Modi and Kovashka
(2017); Plummer et al. (2019). The main idea is that users manipulate
attributes of a query, and the search engine finds a fashion item with the
desired attributes (_e.g_ green striped dress). Although attribute
manipulation is particularly useful in fashion domain, we found that only a
few papers have addressed this issue Zhao et al. (2017); Ak et al. (2018a).
Zhao et al. Zhao et al. (2017) proposed to use a memory block to obtain a
manipulated feature. The feature was combined with an attribute-specific
representation from the memory block. Triplet pairs were constructed based on
a combination of attribute labels, which requires a dense annotation for
attributes. Ak et al. Ak et al. (2018a) suggested a region-aware attribute
manipulation by the integration of the attribute activation maps. The
extracted features per regions are then combined to construct the global
representation.

While aforementioned methods achieved competitive results on their tasks, more
fundamental problems has not been solved. First, despite the powerful
performance of the FIR methods (instance-based, and attribute-based), the
methods were not designed to search for items with desired attributes.
Therefore, the FIR methods have no way to find users similarly-looking items
with the desired attributes, which is a huge disadvantage particularly in
fashion domain. Second, the methods for attribute manipulation enable search
by changing the attributes, but the image representation of these methods is
not specialized for the FIR task. Therefore the FAM methods show worse
performance in finding the same instance items. To remedy this, we propose a
method that can perform attribute manipulation without sacrificing the
retrieval performance by manipulating queries at the feature level.

Figure 2: Top-3 retrieval results after the query attribute manipulation. The
green-bordered images are the query, and the red-bordered images are retrieved
images from the gallery. The first and second columns are the results on NFD,
and the third and fourth columns are the results on DARN. From top to bottom,
each row targets to manipulate shape, color, and pattern attribute,
respectively.

##  3 Feature-level Attribute Manipulation (FLAM)

Our objective is to train a network that converts the feature of the given
query into the feature with the desired attributes. We refer to it as feature-
level attribute manipulation, because the attribute of a learned feature is
manipulated directly at the feature level, and no representation learning from
the image is needed. The feature-level attribute manipulation has several
advantages. First, since the features learned by FIR methods are used as the
input for the manipulation, the quality of the feature performing FIR can be
perfectly preserved. Second, the learned features include high-level
information comparing to the images in RGB space, and the network can be
trained with a smaller number of data, even with the sparsely labeled
attribute information in the fashion datasets. For manipulating the attributes
at the feature level, our main idea is to synthesize the manipulated feature
with a generative adversarial network Goodfellow et al. (2014), by simply
matching the distribution of the manipulated feature with the distribution of
the real data. We split the training into two stages as described in Figure 1:
(1) embedding stage and (2) manipulating stage. We explain the training
procedure of each stage in detail, followed by the way of synthesizing the
manipulated feature that represents the desired attributes.

###  3.1 Attribute-specific Embedder

At the embedding stage, we aim to embed a learned feature to an attribute-
specific embedding space defined as ${e\in R^{k}}$, where $k$ is the dimension
of the embedding space Vasileva et al. (2018). We assume that the attribute-
specific information (such as shape, color, and pattern) is successfully
encoded in a learned feature when it is learned by FIR methods. For a learned
feature $x$, we find the embedding vectors of target attribute $a$ as
$e_{a}=\phi_{a}(x;\theta)$. The goal is to learn parameter $\theta$ of mapping
$\phi$ such that embedding vectors $e_{a}$ and $e^{+}_{a}$ extracted from
samples of the same attribute class $i$ are projected into as close location
as possible in the embedding space and in far distance from $e^{-}_{a}$ of a
different attribute class. We use the embedding layer, which stores the fixed
size of dictionary vector $d_{a}$. The general form of the triplet loss is
formulated as (1)

$\ell(f,f^{+},f^{-})=max\\{0,dist(f,f^{+})-dist(f,f^{-})+\mu\\},$ (1)

where $\mu$ is a margin, $dist$ is a cosine similarity distance. Finally, we
use a triplet loss using both $e_{a}$ and $d_{a}$ as anchors for training
$\phi$. The final loss is formulated as (2),

$\mathcal{L}_{emb}=\ell(d_{a},e^{+}_{a},e^{-}_{a})+\ell(e_{a},d^{+}_{a},d^{-}_{a}),$
(2)

where - symbol means that the vector is extracted from the sample of a
different attribute class compared with the anchor. Multiple attribute-
specific embedders $\phi$ are trained are trained for every single attribute
type, and they work as a teacher of a discriminator by transferring knowledge
in the following stage.

###  3.2 Query Feature Manipulator

At the manipulating stage, our main interest is to synthesize a manipulated
feature $\tilde{x}$ which follows the distribution of real features $x$. To
avoid ambiguity, we define the target attribute type to manipulate as $a$
(_e.g_ color), and the remaining attribute types to preserve as
$\bar{a}_{1},...,\bar{a}_{n}$ (_e.g_ shape and pattern). The embedding vectors
are defined as $e_{a}$ and $e_{\bar{a}_{1}},...,e_{\bar{a}_{n}}$ the target
and remaining attribute types, respectively. Given a pair of learned features
($x$, $x^{-}$) sampled from different attribute classes of $a$, our goal is to
train a generator $G(x,\phi_{a}(x^{-}))\rightarrow\tilde{x}$ that converts $x$
into an attribute-manipulated feature $\tilde{x}$ that represents the target
attribute of - while preserving the remaining attributes of $x$.

Adversarial loss. To make $\tilde{x}$ synthesized by $G$ follow the
distribution of real features, we adopt an adversarial loss Goodfellow et al.
(2014) as defined

$\mathcal{L}_{adv}=\mathbb{E}_{x}[\log
D(x)]+\mathbb{E}_{x,x^{-}}[\log(1-G(x,e^{-}_{a}))],$ (3)

where $G$ synthesizes $\tilde{x}=G(x,e^{-}_{a})$ conditioned on query feature
$x$ and attribute-specific embedding vector $e^{-}_{a}=\phi_{a}(x^{-})$. We
concatenate $x$ and $e^{-}_{a}$ to feed $G$. The adversarial loss forces the
generator $G$ to mimic the real data distribution by playing a minimax game.

Feature matching loss. We aim $\tilde{x}$ to have desired attribute class of
$a$, not affecting the remaining attributes $\bar{a}_{1},...,\bar{a}_{n}$. Our
strategy is to transfer the attribute-specific knowledge learned by $\phi$ by
matching outputs of discriminator $D$ with the embedding vectors Salimans et
al. (2016). To achieve this condition, the last layer of $D$ is split into two
branches, and each branch produces 1-dimensional vector to discriminate
between real and fake, and $(n\times k)$-dimensional vector for feature
matching with $e$, where $e\in R^{k}$. Letting $f(x)$ denote the outputs of
$D$ for feature matching, for given attribute-specific embedders $\phi_{a}$
and $\phi_{\bar{a_{1}},...,\bar{a_{n}}}$, our objective is defined as

$\begin{split}\mathcal{L}_{match}=\mathbb{E}_{x,\tilde{x}}[\lVert
f(x)-[\\{\phi_{a}(x)\\}_{target}\oplus\\{\phi_{\bar{a_{1}}}(x)\oplus,...,\phi_{\bar{a_{n}}}(x)\\}_{remain}]\lVert]_{real}\\\
+\mathbb{E}_{x,x^{-},\tilde{x}}[\lVert
f(\tilde{x})-[\\{\phi_{a}(x^{-})\\}_{target}\oplus\\{\phi_{\bar{a_{1}}}(x)\oplus,...,\phi_{\bar{a_{n}}}(x)\\}_{remain}]\lVert]_{fake},\end{split}$
(4)

where $\oplus$ is a concatenation operation. The feature matching term for the
remaining attributes can be omitted, but we found it harmful for the
performance, as shown in Table 3.

Cycle consistency loss. Recently, a cycle consistency has proved its
effectiveness in preserving the content of the original input Kim et al.
(2017); Zhu et al. (2017). To preserve unique characteristics of a query item,
we apply the cycle-consistency loss to the generator, defined as

$\mathcal{L}_{cycle}=\mathbb{E}_{x,x^{-}}[\lVert x-G(G(x,e^{-}),e)\lVert]$ (5)

An interesting point is that a cosine similarity between original feature $x$
and recovered feature $\hat{x}=G(G(x,e^{-}),e)$ can be used for an approximate
measure of convergence Berthelot et al. (2017). We found a consistent tendency
that a model showing a higher cosine similarity between $x$ and $\hat{x}$
achieves a better top-k accuracy in the attribute manipulation task.

###  3.3 Query Manipulation

To search with a query after the attribute manipulation, a manipulated query
is obtained by $G(x,d_{a_{i}})\rightarrow\tilde{x}$, where $d$ is an
attribute-specific dictionary vector of $i$-th class of attribute type $a$
which has been learned at embedding stage. The attribute-specific dictionary
vector $d_{a_{i}}$ can be considered as a centroid of a feature cluster of
$i$-th attribute class. Finally, a query having the desired attributes can be
synthesized by choosing $d$ of a desired attribute class $i$.

##  4 Experiments

In this section, we describe the datasets, evaluation metrics, and
implementation details followed by quantitative and qualitative results. In
this study, we target three most representative fashion attribute types,
$a\in\\{shape,color,pattern\\}$, for attribute manipulation. When we target a
specific attribute, we call it the target attribute (_e.g_ color), and the
others as the remaining attributes (_e.g_ shape and pattern), for clarity.

Datasets. Although many fashion-related datasets have been released, we found
few fashion datasets provide reliable attribute labels. We used DARN and
Shopping100k to evaluate and train the attribute-specific embedder. The top-k
classes in each attribute type, sorted by the number of images included, were
used to avoid the long-tail distribution. The detailed classes and labels of
the datasets are described in Table 1. To evaluate FAM, we randomly sampled 5K
and 2K images from DARN and Shopping100k, respectively. We only picked the
images labeled with all three attributes densely. The remaining images were
used for the training, because FLAM does not require a dense annotation. To
manipulate the training query features, we used private datasets, Naver
Fashion Dataset (NFD) nav . The NFD is a large-scale fashion dataset collected
from the Naver shopping pages, which includes more than 390K instances of
4-million high-resolution images. The instance pair label was annotated by the
human experts to ensure its reliability. While NFD provides no attribute
labels, we could use the entire images of the NFD by using the attribute-
specific embedder as a pseudo-labeler by means of feature matching.

Attribute types | Top-10 classes (DARN)
---|---
Shape | sweater, hoodies, … , cotton_clothes (clothes_category)
Color | black, red, … , rose_red (clothes_color)
Pattern | solid_color, flower, … , abstract (clothes_pattern)
Attribute types | Top-10 classes (Shopping100k)
Shape | T-shirt, Shirt, … , Tracksuit_Bottoms (category)
Color | Black, Navy, … , Olive (color)
Pattern | Plain, Print, … , Colourful (pattern)
Table 1: Top-10 attribute classes used for experiment. The label in bracket is
the original label in the datasets.

Evaluation metrics. For the quantitative experiments, we define the evaluation
metrics for both FIR and FAM as follows. For FIR task, recall(R@k) Gordo et
al. (2017) was measured. For FAM task, top-k accuracy(T@k) Huang et al. (2015)
was measured. To measure top-k accuracy, the search system retrieves top-k
matches after performing the attribute manipulation on a given query. If a
match is found that has the same attribute combination (shape, color, and
pattern) as the manipulated query, we count it as a hit, otherwise a miss.

###  4.1 Implementation Details

When training an attribute-specific embedder, we set the dimension $k$ of the
embedding space to 32. By squeezing the attribute-related representation in
fewer dimensions, we prevent the embedding vector from including information
irrelevant to the target attribute. When training a query feature manipulator,
$G$, a randomly chosen embedding vector $e^{-}$ is given to synthesize a
manipulated feature. However, sometimes it becomes a problem when attributes
of a fashion item are highly entangled. For example, if a query item is a
brown leather jacket and the embedding vector represents “pink”, there barely
exists such cloth in the real world. In this case, a manipulator is confused,
because it has never seen such data in the real distribution. To avoid this,
we used a technique called online sampling(OS). In detail, for a given batch,
$x^{-}$ is sampled to have as close distance in the embedding space for
$\bar{a_{1}},...,\bar{a_{n}}$ as possible. By choosing $x^{-}$ among the
samples having the same remaining attributes as $x$ as possible, we could
avoid forcing the manipulator to synthesize unrealistic features. Note that a
pair $(x,x^{-})$ is sampled to have different target attribute classes for
$a$. The effect of online sampling is shown in Table 3.

###  4.2 Quantitative Results

In this section, we report the performance evaluation on both FIR and FAM,
comparing them to the baseline methods. The evaluation is performed using the
public datasets: DeepFashion, DARN, and Shopping100k.

Method | DeepFashion(In-shop) | DeepFashion(C2S)
---|---|---
R@1 | R@5 | R@20 | R@50 | R@1 | R@5 | R@20 | R@50
WTBI Chen et al. (2012b) | 0.347 | 0.424 | 0.506 | 0.541 | 0.024 | 0.035 | 0.063 | 0.087
DARN Huang et al. (2015) | 0.381 | 0.547 | 0.675 | 0.716 | 0.036 | 0.063 | 0.111 | 0.152
FashionNet Liu et al. (2016) | 0.533 | 0.673 | 0.764 | 0.793 | 0.073 | 0.121 | 0.188 | 0.226
FashionNet+Poselets Yang and Ramanan (2011) | 0.424 | 0.600 | 0.699 | 0.736 | 0.043 | 0.079 | 0.139 | 0.182
FashionNet+Joints Yang and Ramanan (2011) | 0.414 | 0.605 | 0.682 | 0.723 | 0.044 | 0.075 | 0.136 | 0.180
Ours(ResNet50-SPoC) | 0.887 | 0.961 | 0.984 | 0.991 | 0.265 | 0.497 | 0.664 | 0.755
Table 2: Instance-level image retrieval performance (R@k) on DeepFashion
dataset. C2S indicates the consumer-to-shop subset of DeepFashion.

Baselines. Because the previous works on attribute manipulation required a
dense annotation, and even the target attributes were different, it is unfair
to compare the methods directly. Furthermore, no paper so far has discussed
feature-level attribute manipulation. Instead, we propose the following
baselines to evaluate our method. (1) Cls-Concat: a concatenation of features
from three classifiers for shape, color, and pattern. The network has two
fully connected layers followed by a ReLU and a dropout Srivastava et al.
(2014), and trained by cross-entropy loss. We fixed the size of the
intermediate layer to 341. (2) Emb-Concat: a concatenation of features from
three attribute-specific embedders for shape, color, and pattern. The triplet
loss is used, and the size of the embedding vectors is fixed to 341. The final
dimension after the concatenation in both baselines is 1023 ($341\times 3$),
which is similar to FLAM. We consider these baselines as an upper bound in
performance for FAM.

FIR performance. Because the FLAM performs the attribute manipulation at a
feature level, learning a good representation is important. We used cross-
entropy loss and triplet loss together using ResNet50 He et al. (2016) as a
backbone to train the feature extractor. We sampled thousand instances sorted
by the number of images included in instance id to apply the cross-entropy
loss, and then entire images were used when back-propagating with triplet
loss. The feature maps are pooled using the sum-pooled convolutional features
(SPoC) Babenko and Lempitsky (2015). Note that the FIR performance can be
improved by a careful aggregation of the pooling methods Jun et al. (2019),
although we omitted the experiments on it. By training with the instance
labels only, we could obtain competitive recall on DeepFashion dataset in
Table 2. The FLAM can be seen as a postprocessing method of a learned feature
for the attribute manipulation. We consider the FIR result of our feature
extractor as the performance of the FLAM on FIR.

Method | Dim |  DARN Huang et al. (2015) (T@50) |  Shopping100k Ak et al. (2018b) (T@50)
---|---|---|---
S | C | P | All | S | C | P | All
AMNet* Zhao et al. (2017) | n/a | - | - | - | 0.668 | - | - | - | 0.711
FashionSearchNet* Ak et al. (2018a) | n/a | - | - | - | - | - | - | - | 0.732
Cls-Concat | 1023 | 0.896 | 0.947 | 0.698 | 0.898 | 0.886 | 0.895 | 0.802 | 0.874
Emb-Concat | 1023 | 0.758 | 0.769 | 0.888 | 0.779 | 0.916 | 0.925 | 0.893 | 0.915
FLAM(S/-/Adv) | 1024 | 0.472 | 0.560 | 0.375 | 0.505 | 0.517 | 0.338 | 0.430 | 0.424
FLAM(M/-/-) | 1024 | 0.612 | 0.874 | 0.500 | 0.730 | 0.602 | 0.587 | 0.483 | 0.573
FLAM(M/-/Adv) | 1024 | 0.799 | 0.872 | 0.521 | 0.802 | 0.640 | 0.749 | 0.628 | 0.685
FLAM(M/OS/Adv) | 1024 | 0.824 | 0.867 | 0.624 | 0.822 | 0.633 | 0.768 | 0.633 | 0.691
Table 3: Top-50 accuracy for FAM. The notation indicates M/S: feature matching with/without remaining attributes, Adv: adversarial loss, and OS: online sampling. S, C, and P stand for Shape, Color and Pattern. * means that the value should be used for reference only, because the experiments for these methods used different combinations of attributes.  Method | Dim |  FIR(DeepFashion(C2S) Liu et al. (2016)) |  FAM(DARN Huang et al. (2015))
---|---|---|---
R@1 | R@5 | R@20 | R@100 | T@1 | T@5 | T@20 | T@100
Cls-Concat | 1023 | 0.043 | 0.092 | 0.164 | 0.301 | 0.284 | 0.660 | 0.838 | 0.932
Emb-Concat | 1023 | 0.020 | 0.043 | 0.083 | 0.180 | 0.340 | 0.632 | 0.732 | 0.812
FLAM(M/OS/Adv) | 1024 | 0.095 | 0.200 | 0.329 | 0.519 | 0.176 | 0.458 | 0.707 | 0.884
Table 4: Simultaneous comparison between FIR and FAM performances.

Backbone | R@20 | T@20
---|---|---
ResNet50 He et al. (2016) | 0.235 | 0.733
SE-ResNet50 Hu et al. (2018) | 0.205 | 0.744
SE-ResNeXt(32x4d) Hu et al. (2018); Xie et al. (2017) | 0.240 | 0.780
InceptionResNetV2 Szegedy et al. (2017) | 0.218 | 0.719
DenseNet-121 Huang et al. (2017) | 0.255 | 0.691

DARN Huang et al. (2015) (Classification accuracy)
---
query | S | C | P | Avg.diff
original | 0.352 | 0.730 | 0.768 | -
manipulated | 0.577 | 0.866 | 0.714 | +0.102
Shopping100k Ak et al. (2018b) (Classification accuracy)
query | S | C | P | Avg.diff
original | 0.819 | 0.675 | 0.814 | -
manipulated | 0.843 | 0.941 | 0.749 | +0.075

Table 5: R@k and T@k on different backbones (k=20) and classification accuracy
after the attribute manipulation. The evaluation dataset is the same as Table
4.

FAM performance. The Table 3 reports the top-50 accuracy of baselines and our
method. Compared to the baselines, FLAM shows comparable performance on both
datasets, despite that the query feature manipulator of FLAM was trained in a
semi-supervised manner. We also achieved better results when auxiliary
embedders of the remaining attributes and the adversarial loss were both used.
Without the adversarial loss, the cosine similarity between the original
feature and the recovered feature is low, which indicates that the manipulator
suffers from recovering distribution of the original data. The use of online
sampling technique consistently improved the performance. In Table 4, we
compare FIR and FAM simultaneously, as our purpose is to obtain high
performance in both tasks. As we expected, our method shows outperforming
result in FIR task, as the learned feature was trained for it.

Quality of features, and robustness to backbones. The quality of the
manipulated features and the robustness depending on different backbones were
investigated in Table 5. We trained a simple classifier, and measured the
classification accuracy on the original features. Then, the query was
manipulated, and the accuracy on manipulated feature was measured. The result
shows that the accuracy increases after the attribute manipulation, which
means that the query attribute was well-manipulated as we intended. Moreover,
FLAM shows consistent results on both FIR and FAM, regardless of the backbone,
which shows the robustness of our method in the network structure.

Figure 3: t-SNE visualization of the attribute-specific embedding vectors on
the embedding space. Among 3K samples of the original feature, the color
attribute of randomly sampled 1K features is manipulated to represent a
specific color. The color of the mark represents the original attribute
classes on attribute types.

###  4.3 Qualitative Results

In this section, we visualize the top-k ranking results after the attribute
manipulation in Figure 2, and t-SNE Maaten and Hinton (2008) visualization of
attribute-specific embedding vectors in Figure 3.
Top-k ranking result. The Figure 2 shows the top-3 ranking results after the
query attribute manipulation. The results show that FLAM retrieves the items
having the desired attribute as we intended, while maintaining the unique
characteristics of the query that are hard to define with shape, color, and
pattern only. For example, if the query is a purple-colored slim-fit padding
jumper with fur, and we manipulate color, the retrieved items maintain the
attributes “slim-fit” and “fur”, which have not been learned. We report top-3
ranking results, because we found that the evaluation metric of top-k accuracy
on shape, color, and pattern have a limitation in reflecting the
aforementioned aspect.
t-SNE visualization. To analyze our method in the feature space, we visualize
the distribution of the attribute-specific embedding vectors of shape, color,
and pattern attributes using t-SNE. We manipulate the query features to have
the same color, while maintaining the remaining attributes (shape and
pattern). After manipulating the color attribute, the manipulated feature is
fed to the attribute-specific embedder once again to obtain the embedding
vector of the manipulated feature. The Figure 3 shows that the color
distribution of the manipulated features is focused on a certain location,
which indicates that the color is successfully manipulated to represent a
certain color. The distribution of shape and pattern attributes shows a
similar distribution with one of the original features, which means that the
remaining attributes are preserved while manipulating the color.

##  5 Conclusions

In this paper, we have addressed the problem of attribute manipulation at the
feature level. Unlike previous approaches, feature-level attribute
manipulation (FLAM) translates the attributes from a learned feature itself,
which makes the attribute manipulation process completely independent from
image-based representation learning. We demonstrated competitive results on
both FIR and FAM tasks with FLAM, and showed its robustness in application of
different backbones. We showed that the generative model trained in a semi-
supervised manner can be used for attribute manipulation, by simply matching
the distribution of the manipulated features with the real features.

## References

  * (1) https://shopping.naver.com/.
  * Ak et al. (2018a) Kenan E Ak, Ashraf A Kassim, Joo Hwee Lim, and Jo Yew Tham.  Learning attribute representations with localization for flexible fashion search.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7708–7717, 2018a.
  * Ak et al. (2018b) Kenan E Ak, Joo Hwee Lim, Jo Yew Tham, and Ashraf A Kassim.  Efficient multi-attribute similarity learning towards attribute-based fashion search.  In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1671–1679. IEEE, 2018b.
  * Babenko and Lempitsky (2015) Artem Babenko and Victor Lempitsky.  Aggregating deep convolutional features for image retrieval.  _arXiv preprint arXiv:1510.07493_ , 2015.
  * Berthelot et al. (2017) David Berthelot, Thomas Schumm, and Luke Metz.  Began: Boundary equilibrium generative adversarial networks.  _arXiv preprint arXiv:1703.10717_ , 2017.
  * Chen et al. (2012a) Huizhong Chen, Andrew Gallagher, and Bernd Girod.  Describing clothing by semantic attributes.  In _European Conference on Computer Vision (ECCV)_ , pages 609–623. Springer, 2012a.
  * Chen et al. (2012b) Huizhong Chen, Andrew Gallagher, and Bernd Girod.  Describing clothing by semantic attributes.  In _European conference on computer vision_ , pages 609–623. Springer, 2012b.
  * Corbiere et al. (2017) Charles Corbiere, Hedi Ben-Younes, Alexandre Ramé, and Charles Ollion.  Leveraging weakly annotated data for fashion image retrieval and label prediction.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2268–2274, 2017.
  * Di et al. (2013) Wei Di, Catherine Wah, Anurag Bhardwaj, Robinson Piramuthu, and Neel Sundaresan.  Style finder: Fine-grained clothing style detection and retrieval.  In _IEEE Conference on computer vision and pattern recognition workshops_ , pages 8–13, 2013.
  * Gajic and Baldrich (2018) Bojana Gajic and Ramon Baldrich.  Cross-domain fashion image retrieval.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 1869–1871, 2018.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pages 2672–2680, 2014.
  * Gordo et al. (2017) Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus.  End-to-end learning of deep visual representations for image retrieval.  _International Journal of Computer Vision_ , 124(2):237–254, 2017.
  * Hadi Kiapour et al. (2015) M Hadi Kiapour, Xufeng Han, Svetlana Lazebnik, Alexander C Berg, and Tamara L Berg.  Where to buy it: Matching street clothing photos in online shops.  In _International Conference on Computer Vision (ICCV)_ , pages 3343–3351, 2015.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7132–7141, 2018.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Huang et al. (2015) Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan.  Cross-domain image retrieval with a dual attribute-aware ranking network.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1062–1070, 2015.
  * Jun et al. (2019) HeeJae Jun, ByungSoo Ko, Youngjoon Kim, Insik Kim, and Jongtack Kim.  Combination of multiple global descriptors for image retrieval.  _arXiv preprint arXiv:1903.10663_ , 2019.
  * Kalantidis et al. (2013) Yannis Kalantidis, Lyndon Kennedy, and Li-Jia Li.  Getting the look: clothing recognition and segmentation for automatic product suggestions in everyday photos.  In _Proceedings of the 3rd ACM conference on International conference on multimedia retrieval_ , pages 105–112, 2013.
  * Kiapour et al. (2014) M Hadi Kiapour, Kota Yamaguchi, Alexander C Berg, and Tamara L Berg.  Hipster wars: Discovering elements of fashion styles.  In _European Conference on Computer Vision (ECCV)_ , pages 472–488, 2014.
  * Kim et al. (2017) Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim.  Learning to discover cross-domain relations with generative adversarial networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 1857–1865. JMLR. org, 2017.
  * Kovashka and Grauman (2013) Adriana Kovashka and Kristen Grauman.  Attribute pivots for guiding relevance feedback in image search.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 297–304, 2013.
  * Kovashka et al. (2012) Adriana Kovashka, Devi Parikh, and Kristen Grauman.  Whittlesearch: Image search with relative attribute feedback.  In _Computer Vision and Pattern Recognition (CVPR)_ , pages 2973–2980, 2012.
  * Liu et al. (2012) Si Liu, Zheng Song, Guangcan Liu, Changsheng Xu, Hanqing Lu, and Shuicheng Yan.  Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set.  In _Computer Vision and Pattern Recognition (CVPR)_ , pages 3330–3337. IEEE, 2012.
  * Liu et al. (2016) Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1096–1104, 2016.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(Nov):2579–2605, 2008.
  * Modi and Kovashka (2017) Bhavin Modi and Adriana Kovashka.  Confidence and diversity for active selection of feedback in image retrieval.  In _British Machine Vision Conference (BMVC)_ , 2017.
  * Plummer et al. (2019) Bryan Plummer, Hadi Kiapour, Shuai Zheng, and Robinson Piramuthu.  Give me a hint! navigating image databases using human-in-the-loop feedback.  In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 2048–2057. IEEE, 2019.
  * Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In _Advances in neural information processing systems_ , pages 2234–2242, 2016.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 815–823, 2015.
  * Singh and Lee (2016) Krishna Kumar Singh and Yong Jae Lee.  End-to-end localization and ranking for relative attributes.  In _European Conference on Computer Vision (ECCV)_ , pages 753–769. Springer, 2016.
  * Sohn (2016) Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In _Advances in Neural Information Processing Systems_ , pages 1857–1865, 2016.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ , 15(1):1929–1958, 2014.
  * Szegedy et al. (2017) Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Vasileva et al. (2018) Mariya I Vasileva, Bryan A Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David Forsyth.  Learning type-aware embeddings for fashion compatibility.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 390–405, 2018.
  * Xie et al. (2017) Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * Yamaguchi et al. (2015) Kota Yamaguchi, Takayuki Okatani, Kyoko Sudo, Kazuhiko Murasaki, and Yukinobu Taniguchi.  Mix and match: Joint model for clothing and attribute recognition.  In _British Machine Vision Conference (BMVC)_ , volume 1, page 4, 2015\.
  * Yang and Ramanan (2011) Yi Yang and Deva Ramanan.  Articulated pose estimation with flexible mixtures-of-parts.  In _CVPR 2011_ , pages 1385–1392. IEEE, 2011.
  * Yu et al. (2018) Rui Yu, Zhiyong Dou, Song Bai, Zhaoxiang Zhang, Yongchao Xu, and Xiang Bai.  Hard-aware point-to-set deep metric for person re-identification.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 188–204, 2018.
  * Zhao et al. (2017) Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan.  Memory-augmented attribute manipulation networks for interactive fashion search.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1520–1528, 2017.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2223–2232, 2017.